By Lauren Silva Laughlin; graphics Nicolas Rapp
FORTUNE -- Once upon a time, an ordinary investor -- call him Joe -- would take some of his retirement savings and put it into a giant brand-name mutual fund that advertised in the Sunday paper. The fund would take that money and buy shares in brand-name American companies -- "large caps," the jargon went. Joe would then dutifully put some of his nest egg in a bond fund -- and, maybe, on a flier, invest a small chunk in a tech fund that held pieces of the top 100 names on the Nasdaq. (It was 1999 -- everybody was doing it.)
Individual stocks would rise and fall with abandon on any given day. But the broad market would generally trade in a narrow band: It was rare for the Dow (INDU) or the S&P 500 (SPX) stock indexes to move 3% in any direction in a session. When they did, the news was big enough to merit a banner headline: BLACK MONDAY or ASIAN CONTAGION, for instance. The action was mostly on the Big Board, the New York Stock Exchange, though Nasdaq was sizzling and clamoring for share. The white-shoe firms on Wall Street dominated the action; hedge funds and day traders hovered on the edges.
What a difference a millennium makes!
Today every aspect of this picture has changed in some significant way -- from which financial instruments investors bet on to where they trade them to how fast those trades are executed. (For more, see Carol Loomis's "The ICE Man Cometh.") The first exchange-traded fund came into being in 1993, says the Investment Company Institute. Twenty years later ETFs have drawn $1.5 trillion of your money (though that's still only about one-tenth the size of the mutual fund industry). Just as striking is where investor dollars are going. Today we're putting a robust 17% of our holdings in funds and ETFs that wager on foreign stocks, almost double the share from a decade ago. Even the roster of companies selling and managing mutual funds has changed: Of the 25 largest fund complexes in 1995, 10 are no longer on the list.
Wall Street firms are still titans, of course, minus a few once-storied names (Lehman Brothers, Bear Stearns). But today's market movers are more apt to be bots than bankers. Automated trading systems, following preset algorithms, now trade nearly half the share volume on U.S. exchanges, jostling equities by the microsecond and often sending volatile stocks into a roller-coaster ride. Hedge funds, according to research firm HFR, have grown fivefold in net assets since 1999, to $2.5 trillion, and routinely move markets with aggressive bets.
The story comes through strikingly in the data -- and so, for this year's Investor's Guide, we've told it that way. Here, a graphic look at the stunning change in how we trade.
Enter the ETF
Before 2002, exchange-traded funds were barely a blip on investor radars -- investments in ETFs totaled less than $100 billion, compared with the $7 trillion then held in mutual funds, according to the ICI. The notion of buying and selling baskets of stocks as if they were a single equity caught on, however, and in a big way. The ETF industry has grown more than 10-fold in the past decade, including some $800 billion in net inflows since 2007. At the end of September 2013, investors could choose from nearly 1,300 ETFs, some 15 times the number at the end of 2000.
At the same time U.S. equity mutual funds have fallen well out of favor. Since January 2007, more than $600 billion has flowed out of these plain-vanilla funds. Meanwhile, investors have poured a mammoth $1 trillion into bond funds. And mom-and-pop investors have increasingly looked overseas too. Net cash flows to foreign mutual funds and ETFs have increased by more than $550 billion in the past six years.
Rise of the borg
Over the past few decades, as exchanges have transformed from high-decibel human slam-dances to whisper-quiet electronic billboards, the time it takes to do a single trade has collapsed to nearly zero. And as execution time has fallen from minutes to microseconds, so too have prices come down, enabling traders to jump in and out of stocks both faster and more cheaply. "There has always been a race in financial markets," says James Angel, an associate professor of finance at Georgetown University. He points to Nathan Rothschild's information network in the 1800s, which was set up to receive information about the Battle of Waterloo faster than anyone else, allowing him to profit from the battle's outcome. "It's no different now," Angel says.
With information traveling at nearly the speed of light, there is a physical limit to how fast any trade order can fly over a fiber-optic cable. That fact has forced traders to compete in a new realm: distance from the trading platform. The latest investing arms race, then, is to get an arm's length from an exchange server. Enter a new term in the argot: "Co-location," or renting space for one's trading computers in or near the server room of an exchange. Meanwhile a slew of regulatory and technological changes have wound up giving faster trading firms an edge. One regulation, known informally as Reg NMS, implemented in 2007, helped force the main exchanges to trade more electronically and as a result opened the competition to scores of smaller "cottage" exchanges. Now more than half the trades done in U.S. stocks are put through sources outside the NYSE and Nasdaq.
With more exchanges, computers can race to several places to ping in their trades ahead of others. The aim is to squeak out hundredths of a penny in millionths of a second. And while anyone can buy a machine to do lightning-fast trades and then rent space to nestle it by the exchange servers, a select few trading firms -- such as KCG (formerly Knight Capital) and Citadel -- have managed to develop software that gives them an edge in beating out other trades, says Eric Hunsader, a software developer at Nanex, a research firm in Winnetka, Ill. "That's the secret sauce," he says.
How investors have fared
Throw into the mix Borglike computers trading automatically according to the whims of preset algorithms, and you have a recipe for volatility. Particularly in the years following the financial crisis, turbulence has been a mainstay of 21st-century global financial markets. Between 2008 and 2011 there were 76 days in which the S&P 500 stock index swung over 3% in either direction, more than in the previous three decades combined, according to S&P Capital IQ. On 15 of those days, the market moved more than 8%, an occurrence that seldom happened before the millennium. Those who have sensed some stability in the market in recent months aren't imagining things: Since 2012 there has not been one 3% intraday swing in the S&P. But experts say the structural forces that ushered in the era of volatility -- algorithmic trading and HFT, for example -- haven't gone away. So when the next correction comes, it, too, is likely to come in feverish ups and downs.
With such dizzying changes, you might think investors would have passed out along the way. But even with the shock to the system brought on by the mortgage meltdown and the great panic that followed, most have done well. In aggregate, U.S. mutual funds -- including equity, fixed-income, and other funds -- have edged out the S&P 500 stock index, returning 6.6% yearly on average since 2002, compared with 6% for the S&P, according to Morningstar. Meanwhile, 401(k) balances continue to climb; accounts administered by Vanguard, for instance, hit a record average balance at the end of October, reaching $98,826.
"The toolkit that the industry has given investors is one that seems to work," says Don Phillips, head of investment research at Morningstar. "It used to be that you have huge amounts of money in company stock. Now you end up with a reasonable slug of equities but have a relatively diversified and balanced portfolio." It helps that people who sign up for 401(k)s have rolling contributions. The structure doesn't allow people to make quick decisions and therefore encourages good behavior, say many experts. "We are almost wired to do the wrong thing as an investor," Phillips says. "When the market is up, you want to put in -- and to take money out when the market is down." In a 401(k) you automatically contribute when the market is down -- this helped a lot during the financial crisis. "That's actually where investors bought securities that would appreciate, and it was ones that they'd be least likely to buy had they not been signed up," he says.
That's not to say it hasn't been painful at times. It surely has. Many investors, including those who had to pull money out for retirement in 2008, were hit brutally hard. But on the whole, those who stayed in have found there's one thing about the market that hasn't changed: "Buy and forget" is still a better formula than "Follow the herd."
This story is from the December 23, 2013 issue of Fortune.
The U.S. is transparent but overpriced. What's an individual investor to do?
By Geoff Colvin, senior editor-at-large
FORTUNE -- In a global investment bazaar, where's the best place to invest right now?
Depends on who you are, as three recent responses to that question make clear.
I asked David Rubenstein, co-founder and co-CEO of the giant Carlyle Group (CG) private equity firm (assets under management: $180 billion), where he's looking to buy companies MORENov 26, 2013 8:00 AM ET
The market is up, but Wall Street has less to be excited about now.
FORTUNE -- The stock market probably isn't in a bubble, but there's a bigger risk you need to consider: That buying now will mean much lower returns over the next few years.
The valuation on the S&P 500 is still reasonable enough – a P/E of 16.6, based on trailing earnings, which is only slightly higher than average.
That's MOREStephen Gandel, senior editor - Nov 19, 2013 5:00 AM ET
Manager of world's largest hedge fund says Fed has made the market a risky place to invest.
FORTUNE -- Investors need to get ready for disappointment.
Ray Dalio, who manages the world's largest hedge fund, believes stocks will only return 4% over the next decade. But that might not be bad compared to other assets. Dalio says bond investors will do worse.
The Bridgewater Associates head made the downbeat investment projections on Tuesday MOREStephen Gandel, senior editor - Nov 12, 2013 11:50 AM ET
The market isn't reacting to certain economic news as we might expect, but that doesn't mean it isn't rational.
By Mohamed A. El-Erian
FORTUNE -- So, good news was interpreted as bad news by the markets on Thursday while, on the very next day, good news was indeed good news? Are markets really that fickle? Are convictions really that shallow?
It is tempting to respond yes based on the view that, over MORENov 11, 2013 9:02 AM ET
Erdoes and other fund managers sound off on how to manage money in the shadow of a debt crisis.
By Anne VanderMey, reporter
FORTUNE -- With a last-minute debt deal winding its way through the legislature, the worst of the uncertainty that has roiled global markets in recent weeks may be past. But the upheaval has already undermined confidence in the U.S., even as the economy shows signs of life. Under MOREOct 16, 2013 2:59 PM ET
Shriti Vadera, who was a top economic adviser to former British prime minister Gordon Brown, says no one has the "foggiest idea" how a default would unfold.
By Tory Newmyer, writer
FORTUNE -- Here's a bracingly frank view of our unfolding debt default crisis from across the pond: "It's one of the most shocking things I've ever seen."
That assessment comes courtesy of Shriti Vadera, a top economic adviser to then-British prime MOREOct 15, 2013 5:23 PM ET
What could go wrong on trading desks if the U.S. defaults on its debt? A lot.
FORTUNE -- Government officials used to see it as part of their job to calm the markets. Not anymore. Last week, President Obama and Treasury Secretary Jack Lew both said they thought Wall Street wasn't freaking out enough about the possibility of a debt ceiling default.
Right now, most of Wall Street appears to be betting MOREStephen Gandel, senior editor - Oct 7, 2013 5:00 AM ET
Even in a default, many investors would see Treasuries as safer than stocks.
FORTUNE -- As Congress seems nowhere closer to resolving the nation's budget problems on day four of the government shutdown, federal officials have raised fresh warnings the U.S. could default on its debt.
Credit markets could freeze, the dollar's value could spiral, and U.S. interest rates could skyrocket, the U.S. Treasury Department warned Thursday. If Congress fails to lift MORENin-Hai Tseng, Writer - Oct 4, 2013 5:00 AM ET
Re/Max CEO Margaret Kelly talks about the decision to go ahead with their IPO on day two of the federal government shutdown.
FORTUNE -- Wall Street may be getting jittery as Congress struggles to resolve the nation's budget problems, but investors betting on America's housing market don't seem fazed by Washington's dysfunction.
On the second day of the government shutdown, real estate brokerage Re/Max Holdings (RMAX) made its debut on the New York MORENin-Hai Tseng, Writer - Oct 3, 2013 9:56 AM ET
|Boost for trade as global deal struck|
|Someone bought a $100,000 Tesla with Bitcoins|
|2 million Facebook, Gmail and Twitter passwords stolen in massive hack|
|Five key numbers behind the jobs recovery|
|Economy is improving but why doesn't it feel that way?|