Here Comes You-Know-Who

During the spring of 2000, I didn't sense any great strain in the fabric of the company as I grew accustomed to its rhythms. The basic elements had coalesced: a physical plant, a core engineering team, finance and HR staff, and even marketing in support of a product for which the demand seemed insatiable. The coming months would be about holding on. A previously unmet need was rushing headlong toward the provider of a free solution—bucking our audience numbers higher and higher with each lunge forward. On May 8, 2000, Google's traffic topped eight million searches a day. Two weeks later, it was nine million. In theory, we could grow forever, but each bounding leap threatened to bring our ride to an abrupt and messy end because we couldn't add capacity fast enough.

The biggest jump lay just weeks ahead. No one spoke about it, but as I stood in line at the café, debating what I could actually eat from Charlie's Appalachian Day menu (pickled pigs feet, okra consommé, free-range pork rinds, moon pies with mayonnaise, and Twinkie cheez-dogs), it seemed there were more than the usual number of empty seats. The few engineers I did glimpse hurriedly filled their trays and headed back to their desks wearing stress and fatigue like battle-tattered hockey jerseys.

Rumors and whispers about a big hairy deal had been spreading over the cables and through the cubicles, but no one would confirm whose business we were attempting to capture.

Urs knew. He rode herd on his ops team to build capacity in the data-center cages as fast as humanly possible. We would need every server we could cobble together to feed the ravenous behemoth we hoped to contain there.

We were going after Yahoo.

Inktomi's contract to supply search results to Yahoo was up for renewal in June 2000, and Yahoo did not intend to extend the partnership, a fact they were hiding from the world at large. They wanted Google to provide the fall-through search on their site, just as we did for Netscape. If users couldn't find what they wanted in Yahoo's directory, they would use Google to search the web.

Why the shift? Inktomi saw portal search as an unprofitable sideline—they focused on providing search services for the internal networks of large enterprises—so they didn't feel the need to push themselves on Yahoo's behalf. That opened the door for Google. Larry and Sergey dug deep to offer favorable financial terms,^* and it didn't hurt that the Stanford guys running Yahoo and the Stanford guys running Google had common ground, or that Omid Kordestani knew Udi Manber—the top search guy at Yahoo—or that Google and Yahoo shared a board member in Mike Moritz of Sequoia Capital, a communication channel that smoothed the progress of the deal.

Google also promised dramatic improvements in search quality. Google's technology had surpassed Inktomi's and would continue to do so, because Google did care, truly and deeply, about consumer search. Google, however, wasn't the only contender pushing for a seat at the table to eat Inktomi's lunch.

"FAST^† was a scare for a while in early 2000," Urs admitted. "They came out with a large index and they were pretty fast. They were not bad quality-wise, but they had real trouble keeping their index fresh. Maybe they were trying to do too many things. By 2001 we felt we were clearly better than Inktomi results-wise, clearly better than AltaVista, clearly better than FAST. We had the best search engine."

And what about Google's comparative quality the year before, when Netscape had become a partner? "Netscape was kind of crazy to switch their search to us," Urs confessed. He believed they made the change "in part because they didn't care about search that much ... It was a cost center."

Not to mention that Omid Kordestani happened to be an excellent salesperson. "Omid could type in 'IBM' on Google and type in 'IBM' on AltaVista," Urs recalls, "and say 'Hey look, aren't our results better?' That was the level of sophistication. Our search was good, but our coverage was bad. You had all kinds of queries where we didn't have the page and AltaVista or Inktomi had it. People's expectations were just low."

That wasn't the case with Yahoo, where Udi Manber, a search specialist, was chief scientist.^* "Udi wrote the contract," Urs told me, "so he paid attention to the important things." Manber would settle for nothing less than the best that Google could produce, which he knew to be more than what the search engine was offering at the time.

Larry and Sergey committed to Yahoo that Google would make numerous improvements in a matter of weeks: to set up two new data centers—including one on the East Coast—to freshen the index by crawling more frequently, to reduce spam in results, and to meet strict limits on latency, the time between the search being entered and the results being delivered. Each of these promises would require enormous effort to fulfill, but if all were met, Google would give Yahoo users a visibly better search experience.

Better, but not the best. "What was important," Urs confided to me, "was that if you syndicated something and you had your own property, you wanted to make sure that over time you could innovate and actually have something better than Yahoo search powered by Google. And that was Google search, unrestricted by commercial agreement."

Had the company bitten off more than it could chew? Yahoo's traffic dwarfed Google's,^† and the moment Inktomi stopped answering Yahoo queries, Google would need to respond to each and every one of them quickly and correctly. With the capacity we had in place at the time, it would be like hooking a garden hose to a fire hydrant.

"Operationally it was a large buildup," Urs said, "so we needed to get lots of servers." Five thousand, to be exact, each of which required hand assembly. The new data centers that would hold them had yet to be identified or prepped for occupancy. New networking systems would have to be developed to ensure queries went to the right place and results were identical regardless of where the responding machine resided.

The Google operations team^* worked with the contractor, Rackable Systems,^† building the machines from parts Gerald supplied. A "bat cave" was set up between Charlie's café and the marketing department for the ops staff to test and burn in servers before shipping them off to the East Coast. The loaded racks were large and heavy but fragile. Google tried renting a truck and having ops move the racks themselves, but one fell over in transit and almost crushed a technician. The decision was made to splurge on professionals. That didn't stop the ops guys from tossing hundreds of thousands of dollars' worth of RAM into their cars and driving it all over town.

Once all the racks were ready, they needed to be moved cross-country practically overnight. Christopher Bosch hired a truck to drive nonstop to the new data center in Virginia. It would leave the highway only to change drivers en route, and once it reached the data center, the racks would roll out the back and directly into the new server farm.

Serving capacity was just the first item on a very long checklist, and in some ways the easiest. The pressure to build quickly was enormous, but at least with hardware the task was clearly defined, the process known, and the progress clearly measurable. Adding hardware to add capacity would not solve Google's problems by itself, however, even without the Yahoo deal.

"Our traffic was increasing eight percent a week for a long time," said Jeff Dean. "Any time you have that rate of growth, you basically have to make software improvements continuously because you can't get the hardware deployed fast enough. They were working as hard as they could, but if you add four percent machines per week, and you've got eight percent growth, that's not good. So we continuously worked on improving our serving system and looked at alternative designs for pieces of it to get more throughput." That kind of software required creativity and design breakthroughs that could not be scheduled in the same way as a server assembly line.

"It was actually good for us," said Urs, who believed contractual obligations to continually refresh the index and meet latency guarantees made Google "a more grown-up company" by forcing it to closely monitor its own progress. "We always wanted to be fast, but the contract said, 'The maximum one-hour latency is this.'^* You had to start measuring it. Once you can measure it, it's much easier to set goals and say, 'Can we make this ten percent better by next month?' So a lot of these things started then."