The U.S. economy is in dire need of faster growth, Larry Summers said.

Summers, the director of President Obama's National Economic Council, said Wednesday at the Council on Foreign Relations that the speed at which the U.S. economy grows will make the difference between solving many of our problems and letting them become almost unbearable.

Says it's time to rebuild

"The most important question is does the growth rate pick up significantly in the next three years," Summers said. "If so, a large number of problems will fade away. If not, it will be close to unaddressable."

Summers made the comments as he approaches the end of a two-year run as the president's top economic adviser. The administration has been under pressure for the weak performance of the U.S. economy, which has seen growth taper off sharply this year and unemployment stay near 10% for 18 months.

Republicans in Congress have been hammering the White House for its failure to bring government spending under control, and Summers conceded that it is crucial to restore fiscal sanity sooner rather than later.

"We are not currently on a path where the public sector's revenue and spending assumptions are properly reconciled," he said. "An appropriate framework for doing  that is terribly important."

But he saved his strongest comments for the need to provide more fiscal stimulus now, when the government can borrow at historically low rates of just over 4% for 30 years, in the name of paving the way for stronger growth.

"We are not doing what we could be doing," he said. He called for more investment in what he called classic infrastructure, such as airports, railways, roads and water plants, and broader infrastructure, which underlies the increasingly information-based economic growth the country is headed for.

"Why should we not be in the top 10 of any measure of connectivity?" Summers asked. "Why should 7-11 use information technology more than doctor's offices?"

He also urged policies that will allow the middle class to share in the successes of innovative American businesses, saying George Eastman's development of film helped buoy the economy of Rochester, N.Y., for decades – in a model that seems to have fallen into disuse.

"Americans still have good ideas, but large, prosperous middle classes aren't the result," he said.