For a while, Mr. Simons traded like most everyone else, relying on intuition and old-fashioned research. But the ups and downs left him sick to his stomach. Mr. Simons recruited renowned mathematicians and his results improved, but the partnerships eventually crumbled amid sudden losses and unexpected acrimony. Returns at his hedge fund were so awful he had to halt its trading and employees worried he’d close the business.

Today, Mr. Simons is considered the most successful money maker in the history of modern finance. Since 1988, his flagship Medallion fund has generated average annual returns of 66% before charging hefty investor fees—39% after fees—racking up trading gains of more than $100 billion. No one in the investment world comes close. Warren Buffett, George Soros, Peter Lynch, Steve Cohen, and Ray Dalio all fall short.

A radical investing style was behind Mr. Simons’s rise. He built computer programs to digest torrents of market information and select ideal trades, an approach aimed at removing emotion and instinct from the investment process. Mr. Simons and colleagues at his firm, Renaissance Technologies LLC, sorted data and built sophisticated predictive algorithms—years before Mark Zuckerberg and his peers in Silicon Valley began grade school.

“If we have enough data, I know we can make predictions,” Simons told a colleague.

Mr. Simons amassed a $23 billion fortune—enough to wield influence in the worlds of politics, science, education, and philanthropy. Robert Mercer, a Renaissance executive who helped the firm achieve some of its most remarkable breakthroughs, was a leading supporter of Donald Trump during Mr. Trump’s 2016 presidential election victory.

Mr. Simons both anticipated and inspired a revolution. Today, investors have embraced his mathematical, computer-oriented approach. Quantitative investors are the market’s largest players, controlling 31% of stock trading, according to the Tabb Group, a research firm. Just 15% of stock trading is done by “fundamental” stock traders, according to JPMorgan Chase & Co., as many forsake once-dependable tactics, such as grilling corporate managers, scrutinizing balance sheets and predicting global economic shifts.

Mr. Simons’s pioneering methods have been embraced in the halls of government, sports stadiums, doctors’ offices, and pretty much everywhere else forecasting is required. M.B.A.s once scoffed at the notion of relying on computer models, confident they could hire coders if they were needed. Today, coders say the same about M.B.A.s, if they think about them at all.

“Jim Simons and Renaissance showed it was possible,” says Dario Villani, a Ph.D. in theoretical physics who manages a quantitative hedge fund.

Here’s what’s most confounding about Mr. Simons’s success: He and his team shouldn’t have been the ones to master the market. Mr. Simons had only dabbled in trading before reaching middle age and didn’t care much for business. He didn’t even do applied mathematics—he studied theoretical math, the most impractical kind. Most of the mathematicians and scientists he hired knew nothing about investing; some were outright suspicious of capitalism. It is as if a group of tourists, on their first trip to South America, with a few odd-looking tools and meager provisions, discovered El Dorado and proceeded to plunder the golden city, as hardened explorers looked on in frustration.

Just as surprising are the hurdles Mr. Simons and his team overcame, and how close he was to failing in his quest.

The son of a Boston shoe-factory executive, Mr. Simons developed an early love for mathematics and mischief. As a freshman at the Massachusetts Institute of Technology, he liked to fill water pistols with lighter fluid and use cigarette lighters to create homemade flame throwers, once sparking a bonfire in a dormitory bathroom.

Mr. Simons began his career as a popular professor at MIT and Harvard University. During the Cold War, he broke Russian code working for an organization aiding the National Security Agency. At 37, while running Stony Brook University’s math department, he won geometry’s highest honor, cementing his reputation in mathematics. Friends at the time said Mr. Simons’s rugged, craggy features and the glint of mischief in his eyes reminded them of Humphrey Bogart.

When Mr. Simons left Stony Brook in 1978 to launch his trading firm, he was eager for a new challenge and bursting with self-confidence. At the time, some investors and academics saw the markets’ zigs and zags as essentially random, arguing that all possible information already was baked into prices, so only news, which is impossible to predict, can push prices higher or lower. Others believed price shifts reflected efforts by investors to react to and predict economic and corporate news, efforts that sometimes bore fruit.

Mr. Simons developed a unique perspective. He was accustomed to scrutinizing large data sets and detecting order where others saw randomness. Scientists and mathematicians are trained to dig below the surface of the chaotic, natural world to identify simplicity, structure, and even beauty. Mr. Simons concluded that financial prices featured defined patterns, much as the apparent randomness of weather patterns can mask identifiable trends.

 It looks like there’s some structure here, Mr. Simons thought.

He was determined to find it. Mr. Simons told a friend that solving the market’s age-old riddle “would be remarkable.”

Mr. Simons convinced a reserved Stony Brook mathematician named Lenny Baum, whose work helped pave the way for weather prediction, speech-recognition systems and Google’s search engine, to join the firm. They invested about $4 million for themselves and a handful of investors using crude trading models and their own instincts. An early winning streak came to an abrupt end, though, when Mr. Simons began trading bond-futures contracts. Losses topped $1 million and clients grumbled.

Mr. Simons took the downturn hard. One rough day in 1979, a young staffer named Greg Hullender found Mr. Simons lying on a couch in his office.

“Sometimes I look at this and feel I’m just some guy who doesn’t really know what he’s doing,” Mr. Simons said.

Mr. Hullender worried his boss was suicidal.

Mr. Simons escaped his funk, determined to build a high-tech trading system guided by preset algorithms, or step-by-step computer instructions.

“I don’t want to have to worry about the market every minute. I want models that will make money while I sleep,” Mr. Simons told a friend. “A pure system without humans interfering.”

Mr. Simons suspected he’d need reams of historic data so his five-foot-tall, blue-and-white PDP- 11/60 computer could search for persistent and repeating price patterns over time. He bought stacks of books from the World Bank and reels of magnetic tape from various exchanges, amassing data going back decades.

A staffer traveled to Federal Reserve offices in lower Manhattan to record interest-rate histories and other information not yet available electronically. For more recent pricing data, an office manager balanced herself on sofas and chairs in the firm’s library to update graph paper taped high on the office walls. (The arrangement worked until she toppled from her perch, pinching a nerve.)

Eventually, they gathered data going back to the 1700s—ancient stuff that almost no one cared about but Mr. Simons.

“There’s a pattern here; there has to be a pattern,” he insisted.

Mr. Simons and his colleagues developed a system capable of dictating trades in commodity, bond, and currency markets. Live hogs were a component so Mr. Simons named it his “Piggy Basket.” For several months, it scored big profits, trading several million dollars of the firm’s cash.

Then, something unexpected happened in 1979. The system developed an unhealthy appetite for potatoes, shifting two-thirds of its cash into futures contracts on the New York Mercantile Exchange representing millions of pounds of Maine potatoes. Mr. Simons received an urgent call from regulators at the Commodity Futures Trading Commission that he was close to cornering the market for these potatoes.

Mr. Simons stifled a giggle. He hadn’t meant to accumulate so many potatoes—surely, regulators would understand. Somehow, they missed the humor in the misadventure, however. Mr. Simons was forced to close his potato positions, squandering profits. Soon, they were facing trading losses and Mr. Simons had lost confidence in the system. He could see its trades but he wasn’t sure why the model was making them. Maybe a computerized trading model wasn’t the way to go, after all.

The Red Phone

Mr. Simons began buying and selling like most other investors. Most days, he sat in his office, wearing jeans and a golf shirt, staring at his computer screen while digesting the news. Mr. Simons would hold a Merit cigarette in one hand and chew on his cheek, engrossed in thought about the market.

Mr. Simons hired a Parisian to read an obscure French financial newsletter and translate it before others had a chance to act. He consulted an economist named Alan Greenspan who later would become Federal Reserve chair. Mr. Simons set up a red phone that rang whenever urgent financial news broke, so he could be the first to trade. Sometimes, he and Mr. Baum were nowhere to be found when the phone rang, though, so an office manager raced to find them—even in the men’s room, where she’d pound on the door.

“Come back in!” she screamed once. “Wheat’s down 30 points!”

They were making so much money that a quantitative trading style seemed a waste of time. Mr. Simons and Mr. Baum bought gold and other investments that soared. Mr. Simons became worried and sold his gold, scoring profits, but he had trouble convincing Mr. Baum to act.

“Lenny, sell right now.”

“No—the trend will continue,” Mr. Baum responded.

“Sell the F—g gold, Lenny!”

Mr. Baum grudgingly agreed.

After Mr. Baum discovered a native of Colombia who predicted higher coffee prices, he bought—and proved even more resistant to selling, even as his coffee-futures investments plummeted in value. Eventually, Mr. Baum lost so much money he had to ask Mr. Simons to sell his coffee, unable to part with the investments himself. Mr. Baum and Mr. Simons would part ways.

“He had the buy-low part, but he didn’t always have the sell-high part,” Mr. Simons later said about Mr. Baum.

Dr. Ax

Relying on intellect and instinct didn’t seem to work. Mr. Simons refocused on building a computer trading system reliant on mathematical models and algorithms, an approach that might allow him to avoid the emotional ups and downs of traditional investing.

“If you make money, you feel like a genius,” he told a friend. “If you lose, you’re a dope.”

This effort was led by another acclaimed former Stony Brook mathematician, James Ax. The firm’s data trove was riddled with faulty prices, however. They found another former professor named Sandor Straus to scour the data and ensure it was ‘clean,’ a painstaking task almost no one at the time cared to do. Mr. Straus became something of a data obsessive, gathering more information than their computers could process.

Mr. Ax proved a difficult colleague. Around the office, he pushed conspiracy theories, especially those involving the Kennedy assassination. He demanded staffers refer to him as “Dr. Ax,” out of respect for his PhD. (They refused.) Once, he asked the office manager to tell a driver in an adjoining parking lot to move his car because the sun glare was bothering him. (She pretended she couldn’t find the car’s owner.)

The team made money but there were few hints their efforts would lead to anything special. It wasn’t even clear Mr. Simons would keep the trading effort going. When an employee received a job offer from Grumman, a local defense contractor, colleagues supported his decision to bolt. Grumman offered a signing bonus of a free turkey, after all.

The Casino Model

The doubts piled up in the 1980s. By the end of the decade Mr. Simons was on his second marriage and third business partner. Returns at his Medallion hedge fund were so awful he halted its trading and Mr. Ax quit (Mr. Ax died in 2006). Employees worried Mr. Simons would close the business. Computer trading seemed folly. At the time, traders searched for an information advantage—market-moving financial news unavailable to the general public. Famous figures like Mr. Soros and Mr. Lynch divined the direction of investments, financial markets, and global economies. Mr. Simons didn’t have a clue how to estimate cash flows, identify new products, or forecast interest rates.

A new team that included Elwyn Berlekamp, a computer scientist who taught part-time at the University of California, Berkeley, began identifying reliable and repeatable short-term patterns in the market. They shifted to concentrate on this kind of trading, holding positions for just a few days. The idea was to resemble a gambling casino, handling so many daily bets they’d only need to profit from a bit more than half of their wagers.

Another mathematician Mr. Simons had recruited from Stony Brook, Henry Laufer, made important discoveries demonstrating the market’s recurring and overlooked trading sequences. Monday’s price action often followed Friday’s, while Tuesday saw reversions to earlier trends. Medallion began buying late in the day on a Friday if a clear uptrend existed, and sold early Monday, taking advantage of what they called the “weekend effect.” Odd, fleeting patterns in currency and other markets were identified, some barely perceptible to rivals.

Implementing their new short-term, computer-driven approach, Mr. Simons’s team saw big gains. Outsiders scoffed. When Mr. Berlekamp explained the firm’s methods to business students and others at Berkeley, he was mocked.

“We were viewed as flakes with ridiculous ideas,” said Mr. Berlekamp, who died earlier this year, in a 2017 interview. “Colleagues avoided or evaded commenting.”

Then, Medallion scored a gain of 55.9% in 1990, a dramatic improvement on its 4% loss the previous year. The profits were over and above Medallion’s hefty fees—5% of all assets managed and 20 percent of all gains.

In trading, as in mathematics, it is rare to achieve true breakthroughs in midlife. Yet, Mr. Simons was convinced he was on the verge of something special, maybe even historic.

If he could pull it off, Mr. Simons knew he could make millions of dollars, maybe even more, perhaps enough to influence the world beyond Wall Street, which some suspected was his true goal. The sudden success of 1990 made Mr. Simons more enthusiastic about his approach. He wanted to be the one to use math to beat the market.

Mr. Simons called Mr. Berlekamp, sometimes several times a day, with ideas, suggestions and encouragement.

“Next year we should get it up to 80%,” Mr. Simons said one day in 1990 about Medallion, which managed about $40 million at the time.

To Mr. Berlekamp, the figure sounded preposterous. He believed in their emerging style, but wasn’t as sure it could result in huge gains. Most of all, he was sick of Mr. Simons’s incessant calls. Eventually, he made Mr. Simons an offer.

“Jim, if you think we’re going to be up 80%, and I think we can do 30%, you must think the company is worth a lot more than I do,” he said. “Why don’t you buy me out?”

So that is what Mr. Simons did.

“The hell with it, I’m just going to run it myself,” Mr. Simons told a friend.

Mr. Berlekamp, Mr. Ax, and Mr. Baum were gone. Imposing and unexpected obstacles stood in Mr. Simons’s way. But he was convinced he had discovered the perfect way to trade. A revolution had begun.