Publication Date: October 4, 1996

The Engines Behind the Search Engines

By Sue Mellen

The numbers are almost beyond comprehension, even in these days when we've grown used to jet-propelled Internet growth. Marlborough, Mass.-based Lycos, one of the world-leading Internet search engines, now responds to more than 100 million requests monthly. If you are Lycos, with data demands growing at an estimated rate of 2,000 percent annually, how do you ensure that you’ll be there for next month’s 100 million-plus hits? You invest in some Internet insurance: high-powered hardware designed to store and manage massive amounts of data. Indeed, Lycos has taken out two such insurance policies: one with Sun Microsystems, Inc. and the other with Digital Equipment Corp.

Sun servers provide the backbone of the Lycos engine, which is really no surprise. Sun's systems are now resident in the operations of all the top telephone companies and Internet providers, explains Dennis Tsu, director of Internet product marketing for Sun Microsystems, based in Mountain View, Calif. He points out that 15-year-old Sun has an edge over some other manufacturers, having been involved in networking from the days when the technology had just migrated from the realm of government to academia. When the commercial Internet was still a gleam in developers' eyes, Sun was working with universities to provide network links for researchers at remote sites. In fact, most of us have long forgotten that "Sun" was originally an acronym for Stanford University Networking.

"We’re proud of our heritage with the 'Net. We were involved when the 'Net was still an academic project and we've been a part of it ever since," Tsu says.

The Sun workstations that power the Internet range from $5,000 servers probably best suited to the small providers now dotting the landscape to $1 million systems capable of providing the backbone for big-name companies like Lycos and Excite.

But the intense demands on the Lycos system have dictated that the company look for some ultra-powerful backup for the Sun hardware. For that, the company turned east toward Maynard, Mass., and DEC. According to Sarah Garnsey, Lycos marketing communications manager, it was there that the company found just what it needed: DEC 64-bit Alpha systems, with Unix operating systems and StorageWorks advanced storage technology.

"We still have a very good relationship with Sun, but we decided to purchase the DEC systems because they were offering just what we were looking for: outstanding storage capacity," explains Garnsey.

A 64-Bit Solution

"We acknowledge that Sun is still ahead in some of these markets, but we’re way ahead of them in 64-bit technology," says Ken McDonnell, public relations manager for DEC's AlphaServer group. "They’re currently perfecting a system, but we’re already there." McDonnell says that the 64-bit configuration (as opposed to the current gold standard, the 32-bit hardware) combines with a super-speedy chip (300 to 400 MHz) to create "by far the fastest system in the world. This is the right system for the Internet, with its heavy demands for data at high speeds."

McDonnell says that DEC isn’t abandoning the little guys at it moves into the Big Sky country of the Internet largely staked out by Sun. He says that DEC sells a lot of two-CPU systems to smaller companies doing business on the World Wide Web. "These systems provide the high throughput of a mainframe system at a fraction of the price," he says.

The servers in the Alpha line range in price from about $9,000 for a low-end 400 Series model to $300,000 for a top-of-the-line 8400. Although the price tag for Lycos' cluster of Alpha 8400s and other DEC equipment is unavailable, it is certain to have topped the $1 million mark.

As they extol the virtue of their Alphas, the folks in Maynard, Mass., are quick to point out that DEC's relationship with the Internet goes back even further than Sun's.

"DEC has a long history with the 'Net. We were involved with the Internet when it was just being developed by government and education. And most people don’t know that DEC had the first commercial Web site, www.digital.com. We've also been involved in networking for the last 20 years. When you think about it, the Internet is really just a network of networks," says Kathleen Pacyna, marketing manager for DEC's Internet AlphaServers Program.

A Second Tier of Companies

Sun and DEC cast the longest shadows on the Internet, but other companies are laying claim to territory. After a study of various systems, the search engine WebCrawler decided to replace its PC-based system with hardware from Silicon Graphics, Inc. That’s right—until April, WebCrawler's server was powered just by 10 Pentium-powered PCs.

"There are some nice things about PCs; they’re really fast and really cheap. But, with the growth of the Web, we needed much more indexing capacity and the ability to share memory over several processors. It just wasn’t possible anymore to work with the memory capacity of PCs," says Rob Wilend, director of WebCrawler. He and his colleagues opted for SGI systems because of their "scaleability," he says. "We were able to start with a limited system, knowing we could add on."

Other companies moving in the territory include Compaq and Hewlett Packard, whose strategic alliances with the Internet software giants—Netscape and Microsoft—result in built-in solutions for smaller servers.

Sun’s Tsu points out that, currently, average consumers don’t care too much about what powers their Internet providers. "They just want to know that they’ll get the performance they need," he says. But he predicts that as services continue to proliferate across the system, consumers will become more savvy about the equipment behind their companies of choice. "There are already articles in the trade publications about choosing your server based on reliability and equipment. After all, what else distinguishes all of these providers from one another? They all do pretty much the same thing," he says.

If Tsu is right, other Internet providers may be following the lead of JavaNet, a small firm nestled in the foothills of the Berkshires in Northampton, Mass. This Main Street Internet provider and cafe serves a great cup of coffee and something more. Along with your hazelnut brew, you can sign up for the Internet service JavaNet provides, get some support from the technical crew on hand and take a peek at the hardware powering the service. JavaNet, which currently serves the 413 area code, has enshrined its DEC servers—an Alpha Series 4000 and Alpha Series 1000—in a dimly lit room with a glass door designed for customer viewing.

"We’re not using PCs and we want people to know that. We’re proud that, out here in a basically rural market, we have some pretty high-end equipment powering our service. Why not put it right out front, where people can see it?" asks David Epstein, founder of JavaNet.

Sue Mellen writes from Tyngsboro, Mass.

---

DCI's Internet Expo covers a full range of Internet hardware, software and management issues. Please see our latest online brochure. 

---