Rick’s Wireless Networking

The efficiency of a development or debug tool needs to be looked at with the perspective on how it effects the debug cycle.  Put another way the real goal of a compiler is to make the user more productive in building a new product, not just running more compiles.   

To analyze this real efficiency you must look at the overall debug cycle.  The cycle has three components: 1) the debug trial, 2) analyzing the results, and 3) modifying the build for the next trial.

The debug trial 

This is the time that measurements are being taken.  The trial needs to be run until a problem or issue that needs to be fixed occurs.  Sometimes productive execution can continue beyond the finding of the first issue.  Other times the error has an effect on the future execution and renders any continued execution useless.  As a project progresses the length of trials will on average increase as the equipment under development can run longer and longer without a failure.

Analyzing the results

This is the time to analyze the measurements that were taken during the trial.  This portion of the debug cycle is dominated by the think time as the engineer attempts to decipher the results to determine what caused the behavior.  The format and level of the information is critical to making it easy to pinpoint the issues.  In networking development, protocol analyzers translate the millions of bits that have flown by into symbolically decoded packets to greatly ease the analysis. 

Modifying the build for the next trial 

This involves running the build tool, loading the new build into the target design, and setting up the rest of the test configuration for the trial.  The build tool will vary depending upon the kind of engineer and the portion of the design that is being debugged.  Build tools include assemblers, compilers, hardware synthesis, and FPGA synthesis, among others.  The loading of the build into the target design may just involve downloading to the target over a cable of network connection.  It may also involve burning the image into an EPROM or other memory device, or multiple copies for multiple devices.  Lastly there is the setting up other configuration variables of the target system.  This can involve physical positioning, resetting of mechanical components, and re-initializing equipment to a known state.

The time to accomplish the build can vary widely from a few minutes to many hours depending on the complexity.  Typically the times are between 15 minutes and one hour.  Also this time tends to increase as a project proceeds, because the size of the design increases, and the test configuration increases in complexity.

Changing Bottlenecks 

Early in my career as a programmer there was controversy caused by how expensive computer time was.  The computer had limited access and compiles were typically batched together for maximum efficiency.  Some thought that programmers did not need more compute power, they just needed to do more analysis before they submitted another compile.  Today with the low cost of compute power and the expense of engineering time, it seems silly to focus on computer efficiency, instead of engineering productivity.

I once worked on a project to improve the performance of a software linker.  The link times were taking over one hour of CPU time.  With multiple engineers time sharing the minicomputer this was limiting the engineers to one or two debug trials per day.  With the project we were able to reduce the CPU time required for linking by 30 times.  Suddenly the CPU was no longer the bottleneck.  They could then make between 6 to 8 debug trials per day.  The time for the engineers to analyze the results and access to shared resources now dominated.

The early hardware acceleration products in the EDA industry could run a design simulation a thousand times faster than the design simulation run on a workstation.  However, the generation of the netlist to load in to the accelerator, became the bottleneck to productivity.  It could take hours to generate the netlist for a simulation run that would take only minutes.  This severely limited the usefulness of the accelerator.  Understandably as acceleration products evolved the generation of the netlist also was accelerated.

Shared Resources 

Once the tool bottleneck has been reduced or eliminated, there are often shared resources that can end up becoming a bottleneck.  These shared resources can include the lab equipment configuration, a high-performance piece of test equipment, access to a Faraday cage, and the target design hardware.  The allocation of the shared resources may need to be scheduled, and many engineers will work off hours to increase access.

Bust Trials 

Between 10% and 25% of trials are busts.  By this I mean that no valuable feedback to the design is achieved.  The causes for this can include a simple error in the design logic, improperly configured equipment, or outside interference.  Often the reason for the failure is discovered before any data is transmitted.  The equipment may not even be able to initialize.  This causes a quick desperate survey of all the potential culprits.  Sometimes the issue causes a complete bust and a re-build of the design is required.  Other times a partial bust is caused and the trial can be re-run from the beginning without a re-build being required.

An additional problem is that these bust trials often still take up the time of the shared resource.  Sometimes there is another engineer that has been waiting in the wings.  However, if there is not another engineer ready, the resource sits idle for some period of time.

Recommendations 

Given the above, here are a few recommendations:

One, monitor the number of debug trials per day that engineers are able to make, and make sure that they are not being tool bound. 

Two, maximize the information gathered per trial after it has gotten past the bust issues.  This can be accomplished with debug tools that support interactive queries during the debug trial (see post “Basic Debug Tools and beyond”, http://www.chipdesignmag.com/denker/?p=28).

Three, look for features that help avoid additional trials.   In-place editing can avoid having to make another build, and supports exploration that goes beyond what was anticipated before the start of the trial.  Also it is important that the information is presented at the right level (see post “Being on the Level”, http://www.chipdesignmag.com/denker/?p=29). If the engineer always has to stop to analyze, they will be bumped off the shared resource to make way for the next engineer.

Rick Denker

Packet Plus

I recently had a discussion with a development engineer about what makes an ideal development tool.  He responded that he wanted to be able to work at a high level most of the time, but be allowed to dive to lower levels of detail when required.  This highlights the importance of the abstraction level to the effectiveness of a development tool. 

The abstraction levels within a market generally follow the modules or building blocks that are constructed as a market progresses.  As an example in the chip design world the lower levels are transistors, that has been built up to gates, then RTL (register-transfer level) descriptions, then behavioral and higher level descriptions.  Another example is the Seven Layer OSI model that is used throughout the networking industry.  Follow this link to see a description of the OSI model:

http://encyclopedia2.thefreedictionary.com/Seven-layer+OSI+model

Another blog had an interesting post that discusses the progression of abstraction level in software tools. Here is the link: http://softwarejobstofresh.blogspot.com/2007/11/programming-language-to-software.html

To increase their productivity, engineers continually push to work at higher and higher levels of abstraction.  Imagine the difficulty of designing a chip with millions and millions of transistors with tools that work at the transistor level.  It would be a gargantuan task, involving an army of engineers.  This pressure on more productivity for tools has been fueled by the relatively limited supply of engineers and the onward progression of Moore’s Law  (see http://special-sarfunshafi.blogspot.com/2007/11/meeting-man-behind-moores-law.html). 

However, at the higher levels of abstraction certain detailed information is left out, or not readily available.  When something unexpected occurs you often need to dive down a level and look at the more detailed information to determine where the problem is.  It may be in how the abstraction was built (for example, did the compiler create the right code), or it may be a problem that requires the additional detail of a lower level to diagnose.

When a move is made to a new higher abstraction level, the tools typically lag behind.  In software tools when high level languages started being used, the debug tools still worked at the lower level.  The engineer would look at the assembly or machine code generated, and debug using that.  Over time the debug tools improved to where the engineer can debug almost completely at the level of the high level language. 

Using a tool that is at a different level causes situations where the engineer is stuck performing the translation between levels.  This can be tedious, and error prone.  They may even have some additional tools to help with the manual translation such as a hexadecimal calculator.  They may also miss important information because of the shear volume of the data that needs to be scanned at a lower level. 

As an example, in wireless networking there are several lab bench tools that may be used in development depending on the level of the OSI stack that the engineer is working at.  Among the tools and their corresponding level are: protocol analyzer (packet level), software emulator (instruction level), logic analyzer (signal level) and spectrum analyzer (wave level). 

The networking engineer generally wants to work and think at the packet level or above.  Therefore to help the productivity of networking engineers would call for more tools at the packet level and an improved ability to easily move between the different abstraction levels when they need to go lower.

Rick Denker

Packet Plus, Inc.

The basic level of debug tools for most disciplines share some common characteristics.  I call these basic features “PRINTF” level, because of my experiences as a software developer, and anyone who has developed a software program can easily relate.

Before the use of a software debugger, or the even more advanced in-circuit emulator, there was the most basic debug method I call “PRINTF-level” debugging.  I call it that because it involved the insertion of a PRINTF statement or statements into the software code.  (PRINTF was the language statement for a formatted print in the FORTRAN language.) 

You would put the PRINTF statements at key points in your program to check the values of key variables, or document how far the program had gotten.  Although the tool was crude, it provided sufficient access to debug most programs, but not with optimal productivity.

The salient characteristics of PRINTF level debugging are:

  * Significant effort to use

You have to program all the information that you want.  You only get the level of formatting that you are willing to program in.

  * Measurements must be decided before the debug trial

The measurement to be made cannot be changed during the running of the debug trial.

   * Changes in the design

The design is altered to make the measurement.  This can change the timing or size sometimes causing or masking a problem.

The next level beyond PRINTF is to minimize the need to alter the design, and make the tool interactive.   Minimizing the need to alter the design may involve completely eliminating, or making it small and predictable.  Making the tool interactive gives the user the ability to choose what information to gather during a debug trial, and control the execution flow.  These changes make a dramatic improvement in productivity.  Each debug trial with these improvements can take the place of multiple debug trials with a more basic tool.  Also the time setting up the trial may be substantially reduced.

In many industries the initial debug tools have shared the characteristics of PRINTF debugging, then progressed to more interactive options as the market evolved.  For example, a similar progression of tools has occurred in FPGA debugging.

In the networking equipment industry a majority of the debugging falls into the PRINTF category, because of several characteristics.  One, the trials must run in real-time in order to catch the difficult problems.  Two, there are multiple pieces of test and debug equipment that must be coordinated.  Three, the configurations are made of equipment from multiple vendors.  And lastly, the speed of protocol changes makes it difficult for tools to keep up.

In a previous post I called for improved tools for networking engineers (http://www.chipdesignmag.com/denker/?p=11) and moving to interactive tools would be a big improvement.

Rick Denker

Packet Plus, Inc.

How do you create a balance between marketing and engineering?   First set up the two functions each with a different primary focus.  Marketing with a focus on the customer, and engineering with a focus on the technology.

Marketing is responsible for bringing the customer into the product decisions.  This may be through many methods including: customer research, customer visits, reviewing support requests, and reviewing sales results.   They need to also make sure that they do not become a filter.  If certain discussions need a more technical person involved, marketing needs to facilitate that too.

Engineering is responsible for bring technology into product decisions.  What is possible, what is the best way to implement it, what will it cost.  Engineering needs to make sure they provide good data for making the decisions and guard against the options that they favor the options that they want to build.

The amount of overlap depends on the nature of the product and market.  There is significant overlap for most technical products and there is a need for high levels of interaction. 

The best at working this relationship often have a foot in both worlds.  The marketing people often have a technical degree and experience in engineering.  The engineering people have had significant customer interaction. 

Once a product plan has been agreed to, there also need to be guidelines on how to proceed in a changing world.  Few plans can stay the same for more than six months in our constantly changing world.  But there needs to be understanding that keeps the changes from getting out of control.

Marketing must realize that feature changes make engineering less efficient.  The cost of context switching is real, and can dramatically affect the productivity of the engineering team.  If the feature set changes for each customer, then marketing is not doing their job.  If the feature set never changes there is a good chance that marketing is not talking to enough customers.

Engineering must realize that changes in the schedule or feature set, makes the product introduction less optimal.  The timing of an introduction is often targeted to a specific event that will get maximum impact such as a trade show.  There may also be several events that were done before the event as build up or preparation.  Changing the schedule can severely distort these plans and hurt the effectiveness.  The changing of the feature set causes the documentation to be modified at a minimum and at the worst can cause a need to re-position the product, which can change all the marketing materials and the marketing strategy.

Rick Denker

Packet Plus, Inc.

Early in my marketing career I worked with an engineer at Hewlett Packard that liked to talk about the first rule of engineering being “don’t do something stupid”.  First before you judge the rule literally you have to realize how he used the phrase.  This was not stated as not to try new things or take on risks.  He liked to take on risks and push himself.  The other quality that he had was always thinking beyond scope of his specific area to make sure what he did would fit within the broader context of the product. Although it was often stated somewhat as a joke, I never heard him direct the rule at someone else as an insult.  However, he was always actively questioning to make sure he did not run afoul of this rule. 

In another take on the first rule of engineering, recently Andy Grove of Intel in a speech at City College of New York said “the first tenet of engineering is, Always know what problem you’re working on”. (http://www.time.com/time/magazine/article/0,9171,1538622,00.html?iid=chix-sphere)

What these two rules have in common is thinking before doing.  The engineer always needs to be actively thinking about what they are doing, not just blindly implementing something.  I like both these rules, because they are based on the analytical and questioning strength of engineers. 

A natural consequence is that engineering needs to be active in product decisions.  They need to be able to ask why a particular feature makes sense.  This is a positive sign that they are actively thinking about the problem, and trying to make sure they are solving the right problem.

I also believe that engineering must keep aware of the technology options available to completing a solution.  It is another way that engineers need to be actively thinking.  They must spend a portion of their time exploring and learning about the new options and methods to solve a problem. 

The rest of the organization needs to be tolerant and supporting of these behaviors.  They are part of having a robust and creative engineering team.

Also note that it does not mean that engineers alone should make all the product decisions.   In my next post I will explore how to balance the engineering and marketing to make good product decisions.

If anyone has there own version of the first rule of engineering that they like to use, please post them as a response.

Rick Denker

Packet Plus, Inc.

For a long time in my career I believed that the most important function of marketing was to bring the knowledge and understanding of the customer into the company.  This is a critical part of marketing.  However I no longer believe that it is the most important. 

I now firmly believe that the #1 job of marketing is to:

Assess the potential of new markets and to plan the entry into the chosen new markets 

This is consistent with my earlier post, The Marketing Wedge http://www.chipdesignmag.com/denker/?p=13.  The key thought of the Marketing Wedge being that market factors are more important than customer factors, and that customer factors are more important than product factors.

New markets are the key to long-term continued growth and innovation.  Some of the reasons for this are:

- New markets offer the largest potential gains in revenue.  The gains from a new feature to an existing customer base, or addressing a new customer in an existing market are generally much smaller in the long-term.

- New leading edge markets also are characterized by change and innovation.  Participating in these markets will increase your own innovation.  If your growth and innovation are sagging, explore whether you have been resting on the laurels of your current markets, or taking on the challenges of new markets.

- Growth does not necessarily continue forever.  Even the best of markets eventually become saturated, then stagnate and decline.

- There is always the threat of change to current markets, potentially forcing you into a mad scramble for new markets.  Some changes can be foreseen, but a disruptive technological change is almost impossible to predict.

Also remember that new markets take time to develop.  You need to start your entry into a new market years before the new revenue is needed.

Too often within established companies the market and channel choices are already set in stone.  Because of this many in the marketing profession do not get exposed to this aspect of marketing.  However, the increasing pace of change in markets, the increasing complexity of the sales channel options, and the broad diversity of customers make ignoring this #1 job more dangerous to your companies’ long-term prosperity. 

An insightful analysis of this management behavior is detailed in the book titled, The Innovator’s Dilemma by Clayton Christensen. He explains how rational management decisions can cause management to miss market waves caused by discontinuous changes. 

What this means for marketing is that the old model of “the next bench syndrome” which is very incremental approach is even less likely to apply.  That more risks needs to be taken.  It is hard to predict what you will discover in a new market, so flexibility will be needed for success.  Also sales needs to understand that there will be more testing of the knowledge and capabilities of the sales channels.  They will need to be more flexible too.

If your company has not been flexing this new market muscle, it will more likely than not wake up one day to find it has to take a crash course in finding new markets in order to survive.

Rick Denker

Packet Plus™, Inc.

It is important to understand the strengths and weaknesses of the range of wireless test environments.  Test environment refers to the setup or environment into which the device being tested and the test equipment are placed.  The four primary types of environments are: 

Faraday Cages

Faraday cages are usually large, hand-constructed, copper mesh wrapped boxes or rooms.  Because of the expense of their construction, they are typically found in the labs of large equipment manufacturers, where they are shared for testing and quality assurance.  Because Faraday cages assure a fairly noise- and interference-free environment, they are good for a wide variety of individual product tests, especially for antennas.  However, test configurations of more than a few devices can quickly congest traffic in a cage.  In addition, there may not be enough distance in the cage to test effects such as multi-path or diversity. 

Test Boxes, or RF Chambers

RF Chambers are metal boxes with absorbing material lining the inside to dampen interference.  They provide a controlled environment for much lower cost than a Faraday cage.  Typically, the DUT is placed into the test chamber, and probes are used to couple signals to/from the DUT through cables to an external test system.  In some cases the DUT and the test equipment are placed within the same test chamber, at which point this approximates a Faraday cage.  At some point, it ceases to be practical to use chambers as opposed to a Faraday cage.  Moreover, because spatial information is lost, some equipment cannot be tested in a chamber, e.g., smart antennas. 

Multiple sizes of chambers may be required for proper testing in a fully-enclosed environment.  The lower limit on the size of the chamber is dictated by the distance at which the RF near-field transitions to the RF far-field.  Objects – including the walls of the chamber itself – that are placed closer than this distance to the unit under test have a significant impact on the radiation pattern and efficiency of the unit; hence it is necessary to ensure that the chamber dimensions are greater than this distance, otherwise the test results may prove to be either irreproducible or erroneous. 

Cabled

Cabled tests simply substitute a wired connection for the wireless connection, bypassing the antennas and directly connecting two pieces of equipment.  As a result, cabled tests are inexpensive and easy to configure, and provide good isolation from interference.  They are not limited to small configurations, like cages and chambers.  However, because of the lack of interference, their results in configurations are idealized toward better performance than would actually occur in the randomness of an open air environment.  In addition, properly performing cabled testing relies on the DUT itself being well-shielded, which may not always be the case in consumer or low-end enterprise equipment.  In addition, equipment with integral antennas (where the antenna cannot be disconnected to gain access to a connector or other attachment point for a cable) cannot be tested using this method. 

Open air

Open air is the only test environment that truly matches the way the customer will use the equipment.  Like cabled environments, open air has no size limitations or limits on the number of pieces of equipment in a configuration.  For some tests, it is ideal because it can test both the antenna and the protocol effects.  Also it is the only solution for certain location-dependent tests. 

Open air test environments can be separated into indoor and outdoor.  Indoor environments are normally actual buildings, usually with furnishings and other accoutrements characteristic of typical office buildings.  Outdoor environments are usually open spaces without obstructions, such as would be found at an antenna range.  Of these two, the indoor environment is of the most interest, as it most closely approximates the conditions under which the equipment is expected to function.  Outdoor environments are used for applications such as characterizing antenna patterns, setting baselines for range and rate, etc.

Summary of test environments

Complete testing requires a combination of test environments; a one-size-fits-all environment does not exist for wireless testing.  Ideally, test equipment should be able to accommodate all environments.  Click on the thumbnail below to see a summary table.   

Rick Denker

Packet Plus™, Inc

In new markets a bottleneck can develop in the Quality Assurance department.  This post discusses why this occurs and outlines potential solution areas.

The Cost of a Problem 

First it is important to look at how the costs to fix a problem depend significantly on in what phase the problem is discovered.  The conventional wisdom has been that the cost to fix a problem goes up ten times for each stage in of the development process from engineering to the customer. 

This makes intuitive sense.  In engineering a fix may involve a simple re-compile/re-test.  Only one functional group in the company has been affected.  In the best case only a single person is affected.

Once a product has been released to QA there has already been a tremendous investment made in integration, and unit-testing.  In QA there needs to be a test added to the regression to cover the newly discovered error and the previous testing re-run. 

Once a product that has been released to manufacturing the costs again go up dramatically.  The costs can include re-fabrication of a semiconductor device, reworking inventory, replacing inventory, and even re-tooling the manufacturing line in some cases.

Again the costs increase dramatically again once a product is released to customers.  The costs now are both increased expenses and lost revenue.  The expenses include support time handling multiple versions, recall costs, and replacement costs.  However, the impact to customer satisfaction, reputation, and market share may swamp the expenses.  All of these costs are magnified by the size of the installed base.

(Note that all the costs of previous stages will be incurred for issues that are discovered in a later stage.  For example, an error that is discovered by a customer will still have manufacturing costs, QA costs, and engineering costs.)

The Pressure on QA 

This cost structure already puts a lot of pressure on QA to avoid the costs of problems getting to manufacturing or to the customer.  In a new and growing market there are several factors that can combine to create a bottleneck in QA.

Among the factors that increase the load on QA are:  

1) The level of quality that is demanded rises. 

What was acceptable when a product was new and unique becomes unacceptable as more alternatives become available. 

2) The need to test with other products can increase dramatically

As a market grows, this interoperability testing can become a significant part of the QA effort.

3) The average user sophistication becomes lower. 

The early adopters may be experts that can work around certain defects and still get value.  As a market moves into the early majority this will no longer be the case.           

4) The new product may be finding new uses

As the product proliferates or as the price drops, the product may be penetrating new classes of customers.  Suddenly many new application scenarios need to be tested.

Potential Solution Areas 

Clearly from looking at the costs the solution has to come during the engineering or QA phases.  Releasing products prematurely just become ticking time bombs ready to explode at the wrong moment.  Actions such as outsourcing customer support are just stop gap measures that do not address the root cause.

There are many tools that help to automate the QA effort.  These can be invaluable.  They can both decrease the labor costs and increase the effectiveness of testing. 

The best solutions are the ones that can reach all the way to the engineering phase.  They typically involve changes to fundamental processes.   Depending on the kind of issues a company has it could include: getting customer feedback earlier in the process, more unit testing in engineering, eliminating fuzzy handoffs, and clearer release criteria.  As with most organizational changes they can be better implemented when there are tools that support and reinforce the new behaviors that are desired.

Rick Denker

Packet Plus™, Inc.

History 

In my experience I saw that it was much more important to get into the right markets, and learn from the customers, than to try to build the right product for the wrong market.  As I moved into management I found that I had to explain this theory more and more to management in a credible way.  This led me to develop this framework.

I have shared it with many colleagues that have found it useful.  It has been presented as part of a panel to the Portland chapter of the Product Development and Management Association, and to multiple Portland State University classes.

It is similar to some other frameworks for making sure that marketing is covering all the bases.  I find it useful, because it makes sure that marketing does not get lost in the details, and maintains its key role in the directionof the company.

Market, Customer, Product 

I classify marketing activities into three buckets: markets, customers, and products. 

The Market factors are:

            Market size

            Growth rate

            Competitiveness

            Speed

            Trends

            Standards

            Adjacent markets

The Customer factors are:

            Problem to be solved

            Rank versus other customer problems

            Alternatives

            Sales channel

            Reach

The Product factors are:

            Positioning

            Differentiation

            Competencies

            Completeness

            Fit with sales channel

Where to spend Marketing time 

On which set of factors do you spend your time?  The basic premise it that Market factors are much more important that Customer factors, and that Customer factors are much more important than Product factors. 

One way to think of it is the impact a change of one of the factors could have to a business.  A market change generally means a dramatic shift (a 360°change of course).  A customer change means a large shift (a 90°change of course).  Lastly, a product change means a slight shift (a 20° change of course). 

Another way to think about it is that if you get the market and customer factors right, then the customer will give you feedback to correct a product that isn’t quite right.  But if you get either the market or customer factors wrong, then the feedback will not necessarily get you to success.

It helps explain why often the better mousetrap does not lead to a great business.  Often the market and customer factors swamp what is clearly a better product. 

In my experience, it works well for technology markets and I assume that it would work well for many other fast changing industries. 

Rick Denker

Packet Plus™, Inc.

In the networking industry most test tools have become focused on the QA engineer.  I believe this has developed because of two factors:  first, is the need for a complete network set-up running at full speed to uncover the majority of issues.  Test cases that are not full speed often do not uncover any interesting issues.  Second, the issues are not easily uncovered in simulation, either at the chip or network level.  Simulation is often too regular to represent the randomness of actual network traffic. Typically the development engineer is left with only a software protocol analyzer at the packet level, and tools that are focused on other abstraction levels.   

The development analysis tool 

A development tool must be available early in the design process when the development engineer is the most active in debugging a new design.  This means at or before first silicon/first prototype.  This drives a set of needs that are very different than QA.  The needs and the characteristics of development tools are summarized below. 

Needs:           

- work when unsure of many parts of design           

- support broad usage by the development team           

- allow extensive control           

- ability to explore new options           

- work in leading edge environment 

Characteristics:           

- complex interface that may require highly trained user           

- limited size, and easily portable           

- inexpensive so it is economical for the individual engineer 

Tools that have these characteristics may have application in customer support, manufacturing fault analysis, or offloading an expensive QA tool.

The quality assurance tool

The QA tool must provide complete coverage of all the features and combinations of features that make up the design.  This must be able to be run after each new release or update.  This tool does not need to be available as early, but must have more coverage.  This drives the set of needs and characteristics summarized below. 

Needs:           

- support repeatable regression testing           

- coverage of maximum configurations           

- provide an efficient interface 

Characteristics           

- larger, and not easily portable           

- simple easy-to-use interface with pre-written tests and extensive reports           

- limited control            

- expensive, priced for a project or company use 

The development team often needs to use the QA tools when working on larger configuration testing. 

Comparison to other industries 

Interestingly the split between development and QA has evolved differently in other technical disciplines. 

For the software industry, development tools progressed much faster with QA tools being almost non-existent initially.  QA tools came later and still typically lag.  In specialized area such as multi-user loading for internet servers quite advanced QA tools have been developed. 

In the Electronic Design Automation industry was initially dominated by development tools, but shift toward verification now takes up more than half the effort of most chip designs. 

For the hardware industry there has been more of a balance with development have larger staffs and tools budgets, but with QA having the larger and more complex equipment. 

Rick Denker

Packet Plus™, Inc