Highlights
| Slides
WorldWide BenchMark Project Objectives
The 1995 Worldwide Benchmark project was funded by the Industry Canada
with the primary objectives of:
- 1) Determining the competitiveness of Canadian companies in software
development in the industry sectors of aerospace, distribution, financial
services, software, and telecommunications equipment and services.
- 2) Assessing the impact of software development innovations on
software development and support productivity and quality.
- 3) Identifying the critical factors considered by commercial organizations
in selecting geographic areas for "offshore" outsourcing of software
development activities.
Analysis of Software Engineering Data
The findings presented herein (which do not represent the opinions or viewpoint
of Industry Canada) are the result of conducting analyses of software
engineering data obtained from 227 companies worldwide at a detailed level,
360 companies worldwide at a high level, 125 companies in India, and 40
companies making use of offshore service. the overall conclusion of the study
was that Canada is well positioned to be a global force in information
technology. However from a global IT perspective, the most important findings
of the project was that the rate of performance change itself is changing rapidly.
The "bandwidth" of performance as measured by productivity and quality has
expanded from a mere 4:1 about 5 years ago to more than 600:1 as shown in
the study.
The 6 Industry Groups Focused On
In interpreting the findings it is important to know that the project focused on 6
industry groups -- aerospace/defense, distribution, financial services,
software producers, and telecommunication equipment and services
companies. The desired geographic coverage included Canada, Europe, the
United States, and low wage countries (primarily India). The target sample size
for the project was set at 5 to 10 companies per industry group within each
geographic. This provided us with a range of total sample size from 75 to 150
companies.
Acquisition of Data
In terms of methodology, project data collection was performed by
questionnaire and follow-up telephone audit and interview. Areas of software
engineering and organizational level data acquisition included:
| Organizational and demographic data | Productivity and quality data |
| Work profile data | Financial data |
| Tool and technique usage data | Technical infrastructure data |
| Process and certification data | Skills/education and human resource
data |
| Critical issues data | |
| |
| |
The low wage country and outsourcing analysis focused on additional
parameters:
- Critical issues in outsourcing provider selection
- Satisfaction data
Data Demographics
In all, 890 companies were contacted to participate in the detailed software
engineering study, 227 agreed to participate, with the resultant ranking by
"yield" across geographic areas as follows -- India, Far East, England (Europe),
South Africa, Canada, South America, United States. "Yield" in this context
refers to the percent of companies contacted that actually agreed to participate.
In addition, the news of the initiation of this project was made available directly
to major metric "societies" in the U.S. and in Europe. The two organizations in
Europe immediately broadcast the invitation to participate to their membership.
No U.S. group opted to do so. No equivalent Canadian group was identified.
Ranking for Response Rates
The strongest response rate was obtained in the financial services sector. This
was followed in rank by aerospace companies, software producers,
telecommunications companies (equipment and services), distribution
companies, and computer manufacturers.
Supplemental Data Resources
Supplemental data resources for this study included:
- The 1994 Industry Watch Study of organizational IT parameters by Dr.
Howard Rubin -- 360 companies
- The 1994 India IT Practices Study by Ed Yourdon and Dr. Howard
Rubin -- 125 companies
- The Low Wage Outsourcing Study performed as part of this project --
42 responses
Less than 1 in 6 companies could provide defect (quality) data; about 1 in 4
had project level data or basis work profile data; only 1 in 2.5 had an inventory
of language and tool use; and only 1 in 2 had sound budget data. These results
by themselves are indicative of the poor quantitative state of software
engineering practices worldwide. Now on to the findings...
Findings: Work Profile
Work profile findings address the distribution of IT activity across a number of
major categories of work. Worldwide, new development is the most intense
area of activity with an average of 52% of resources being deployed. Canada
has the greatest concentration of work (57%) activity in the "new development"
activity area. However, Canada also has the lowest level of resources
dedicated to systems migration. This is quite interesting in the context that
migration from mainframe to client/server platforms is one of the most intense
areas of activity in the U.S. and other areas. The U.S. shows about 51% of
activity in this area.
Findings: Worldwide
Worldwide, development activity is strongest in the aerospace and distribution
sectors at 59% and 60% of effort. The development "push" in Canada is
strongest in the aerospace and telecommunications sectors and lowest in
financial services. In the U.S., the financial services sector shows a 60% level of
deployment towards new development.
Worldwide, we found that telecommunications companies and software
producers are exhibiting the highest focus on systems migration. In Canada,
software producers are showing the highest level of focus on migration while
telecommunications shows the least. Anecdotal evidence indicates the new
systems development in this sector is targeted at ground-up development of
new systems in the client/server architecture, so the data may not accurately
describe the totally of the situation. The trend in telecommunications activity is
also highest in the U.S.
Maintenance
Maintenance is an area that has been the focus of a massive amount of IT
attention over the past 25 years. Worldwide, financial services organizations
exhibit the highest level of resources deployed in this area ( 40%.) This is also
true in Canada. In the U.S., the aerospace sector shows the highest of all levels,
49%.
All organizations surveyed expect their work profiles across these 3 categories
to shift over the coming year. In particular, it seems that Canada is preparing to
shift to platform migration with significant decreases in both maintenance and
new development. The U.S. in contrast has little focus on decreasing
maintenance but sees an increase in the migration area.
"Drilling Down" Maintenance
"Drilling down" into the maintenance category, the following findings are
apparent:
- The dominant activity is adding new functionality to system with, on a
worldwide basis, fixes/repairs and perfective maintenance consuming an
equal amount of resources
- Canada exhibits the highest concentration of work activity on
fixes/repairs
- Canadian companies are planning a shift that will enable them to lower
the repair burden while increasing their ability to add new functionality
Worldwide, fixes/repairs consumes the highest amount of resources in
the telecommunications sector. (This is also true in Canada and in the
U.S.)
- The new functionality focus is strongest in the distribution industry across
all geographic areas reporting with the exception of Canada where it is
strongest in telecommunications services.
Worldwide, and in the U.S. the aerospace sector has the greatest focus on
perfective activities; in Canada, financial services exhibits the highest activity
level in this area.
"Drilling Down" the Work Profile
"Drilling down" in the new development aspect of the work profile can be done
in the context of the life cycle work distribution. For purposes of this study,
participants were asked to map their internal work breakdown structures to a
generic waterfall-type life cycle. Results are as follows:
- Worldwide, coding and unit testing activities still are the dominant area of
resource concentration
- The U.S. shows the highest level of early life stage emphasis (analysis
and design) and the least on documentation
- In the U.S., aerospace and financial services are the most representative
of this distribution
- Canada exhibits the lowest early life stage emphasis with expected
downstream emphasis on systems and integration test activity
in Canada, financial services and telecommunication equipment
companies show the strongest evidence of this behavior
Findings: Portfolio Support and Language Use
Characteristics
Portfolio support productivity was measured by either function points (FP) per
support professional or thousands-of-lines-of-code (KLOC) per support
professional for purposes of this project. As measured by KLOC/professional
the range of results observed was from 1.5 KLOC/professional to 941
KLOC/professional. This represents a bandwidth of 600:1.
Worldwide average support rates using this metric were approximately 120
KLOC/professional with the U.S. at about 132 KLOC/professional and
Canada, at the lowest average level, of 111 KLOC/professional.
Using FP/professional as a metric based on a separate sample from the
KLOC/professional numbers, the results are as follows:
- The range of observations was roughly 4 FP/professional to 5,900
FP/professional
- Canada exhibited the highest support rate at about 1800
FP/professional.
- The U.S. exhibited the lowest support rate at 950 FP/professional
| Industry Specific and Language Findings Industry specific findings (available in KLOC/professional only) in Canada
show:
- Financial services at the highest average support rate -- 200
KLOC/professional
- Software producers at the lowest support rate -- 30
KLOC/professional
Language use findings indicate:
- Worldwide language mix is quite diverse
- Canada has a strong focus on traditional languages with a fairly narrow
mix
- The U.S. exhibits the most imbalanced mix with many languages at a
low level of usage but a few at high levels.
Findings: New Development Productivity
New development productivity was measured by either function points (FP)
per development professional or thousands-of-lines-of-code (KLOC) per
development professional for purposes of this project.
The top producer as measured by KLOC/professional in this study was
Canada at an average of 14 KLOC/ professional. The average across all
respondents was 12 KLOC/professional. The U.S. at 6.9
KLOC/professional was the lowest overall value obtained. Because of the
availability of historical U.S. data, this finding indicates a drop in overall U.S.
new development productivity as measured by the KLOC/professional
metric. The range in reported performance was again large. The low value
obtained across all samples is .6 KLOC/professional. The high value is 45
KLOC/professional which is indicative of a 70:1 to bandwidth.
KLOC and FP Data
Function point (FP) data was again obtained from a KLOC-independent
sample. The results are as follows:
- Canada again exhibited the highest enterprise level development
productivity at 111 FP/professional
- The U.S. again showed the lowest rate at 88 FP/professional
- The worldwide average is 92.5 FP/professional
As with the KLOC data, this shows a decline in U.S. productivity as
measured in this manner. The ranges of productivity observed exhibited a low
of 42 FP/professional to a high of 170 FP/professional. This is a much
narrower bandwidth than represented in the KLOC data and may be
attributable to the smaller sample size.
Findings: Defect Rates
Defect rates were obtained as a surrogate measure of software quality. Defect
"rates" were measured in terms of defect density as post-release defects per
KLOC or FP. Again independent samples were used to obtain KLOC and
FP data.
The overall comparison in terms of KLOC:
- Canada had the highest overall defect density observed (lowest quality
software) at 5.12 defects/KLOC.
- The U.S. had the lowest overall defect rate observed 2.3
defects/KLOC
- The overall worldwide average defect rate is 3.79 defects/KLOC
Defect Density Data
When defect density data is analyzed by industry the following findings are
evident:
- Telecommunication equipment providers exhibit the highest rate at 5.25
defects/KLOC
- Software producers and telecommunications services providers are the
lowest at roughly 2 defects/KLOC
FP Results
The KLOC results were not reflected in the independent FP results to some
extent:
- Canada again had the lowest defect rate at .03 defects/FP
- The worldwide average was .9 defects/FP
- The U.S. average was .449 defects/FP
Defect densities were highest in the software and telecommunications
industries worldwide (at the .75 defects/FP level). Defect densities were
lowest in the financial services industry at .22 defects/FP.
Little data was available for Canadian industry outside of financial services
and telecommunication services. However both exhibited substantially lower
than average defect rates (.05 defects/FP and .03 defects/FP respectively).
Anecdotally, it seems that Canadian companies that are using FP as a metric
have other characteristics that are indicative of improvements in their software
delivery capabilities.
Findings : Software Process Maturity and ISO 9000
Participants in the 1995 Worldwide Benchmark Project were queried as to
their software process assessment and ISO related activities. All participants
were given a self-assessment tool that applied the guidelines developed by the
Carnegie Mellon Software Engineering Institute in 1987.
The results are as follows:
- 35% of all participants have taken part in some sort of process
assessment
- The highest levels of such activity are outside of the U.S.
- 33% of U.S. participants have been involved in an assessment
- Only 29% of Canadian participants have been involved in an
assessment
- The greatest concentration of activity has been in the aerospace
industry
Self-Assessment Toolset Results
Using the self-assessment toolset, reported results are as follows:
- 74% of all analyses were at Level 1
- 77% of U.S. analyses were at Level 1
- No analyses reported higher than Level 3
- Canada had the highest concentration of Level 2 responses and the
lowest of Level 1 responses
- The aerospace industry in Canada showed the highest overall ratings
ISO 9000 certification was highest in the Canadian sample. This was at the
22% level in contrast to all responses which was at the 18% level and the
U.S. sample at the 2% level. Canada also reported the highest frequency of
organizations planning to move toward ISO 9000 certification, 67%. The
U.S. observation was the lowest, at 6%. The worldwide figure was 20%
across all participants.
Findings: Tool and Technique Usage and Perceived
Value
Worldwide tool and technique usage exhibited the characteristics generally
observed in similar studies -- broad use of a diverse set of tools and
techniques couples with low depth of usage (penetration). The only tools or
techniques reported to be in use beyond the 80% level of the sample is
methodology at the "all response" level and the client/server toolsets which
barely crossed the 80% level. The U.S. is strongest in companies adopting
client/server toolsets. Canada is generally behind in tool adoption except in the
area of methodology.
No tool or technique is reported to be in use by more than 80% of the
professionals that should be applying the tool or technique. In terms of
ranking, networked workstations, reuse of code, and methodology rank high,
Similarly, use of metrics, estimation tools, code generators, object oriented
development, and quality assurance practices rank low.
Findings: Staff Profile and Compensation
Principal findings in this area are:
- Worldwide companies are providing an average of 6-8 training days
per year per staff member
- at the high end companies are providing up to 12 days
- U.S. companies tend to provide the most training; U.S. companies
have the highest number of staff with no formal training for their job
- U.S. companies put the least emphasis on acquiring staff with advanced
degrees
- Canadian companies tend to provide less training for their staff
- Canadian companies have the greatest concentration of staff at the
software engineer level
- Canadian companies make the best use of staff with technical school
degrees
Average Salary Results
The results show that in terms of average salary cost (highest to lowest)
against those surveyed (India excluded) the countries supplying data rank as
follows:
- Switzerland
- United States
- England
- Ireland
- Canada
- Greece
- India, if included would rank in seventh place.
Findings: Budget
Participants were requested to provide IT spending data as expressed as the
percentage of gross revenue spent on IT. The average across all observations
was approximately 7.2%. The Canadian figure was roughly 7% while the U.S.
figure was 3%. The dominant spending category is development and support
(42%) followed by operations (20%), hardware acquisition (13%), software
acquisition (9%) , training (5%), and other expenses (16%). Budget optimism
is quite low. In the U.S., participants expect to see increases of no more than
.25% while in Canada increases on the order of about 2% are expected.
Worldwide average figures go as high as 7.5%.
Findings: Technology Infrastructure and IT Priorities
The mainframe focus of the IT community is evident from the findings of this
project. However, particularly in the U.S. the "push is on" toward the
client/server architecture. This is consistent with the aforementioned work
profile findings. In all areas except Canada, the client/server focus is about to
overtake the mainframe orientation.
In terms of other aspects of infrastructure, particularly that focused on the
development and support community, the networking of developers in
essential. The U.S. leads in this area with slightly more than 94% of
developers being networked in the group surveyed versus a worldwide
average of 88%. Canada is the lowest in this group at the 84% level.
Infrastructure Priorities
Infrastructure priorities are well aligned worldwide. The consensus is that the
priorities at the infrastructure level are improved software process, improved
productivity, decreased cycle time, and improved quality.
At a higher level, the priority picture drawn from the organizational level data
reveals a complementary picture. The top 5 IT issues are :
- improved business alignment
- using IT to reengineer the business
- updating of staff skills
- improved quality
- use of metrics for continuous improvement
On an industry by industry basis these priorities vary to some degree. The
most notable point is that the top 5 list is not truly representative. Issues of
cost and architecture dominate many sectors. In addition, the most notable
departure is in the software industry with its focus on quality.
Synthesis
Software engineering and information technology are clearly at the forefront of
a new arena of global competitiveness. While there still is not a world leader in
this area, some major trends are evident.
In the U.S. for example, major software packages (read this as SAP, for
example) are coming in from other areas of the world. In the meantime the
"piece work" of software engineering, programming, is moving offshore. To
me this scenario seems all too similar to the ones that have already been
played out in the automobile industry --- the influx from Japan and Germany
--- and the textile industry --- how many of your clothes are made in the U.S?
Concluding Thoughts
Opportunities for becoming well positioned in a global sense abound. In my
view, any country desiring to be a player should consider:
- Establish a "pull" for driving software quality up as a national priority
- National certification programs
- National awards
- Provide national industry with "relativity tools"
- Information products that support assessment of competitiveness
- Information products for dissemination of best practices
- Encourage start-ups to attract outside investment
- Software "free trade" zones
- Leverage work force
- Encourage universities to focus on software engineering
- Certify university programs/SEI level 5 model labs
|
|
