Kenneth Fisher, chief executive officer of Fisher Investments, speaks at the Forbes Global CEO Conference in Sydney, Australia, on Tuesday, Sept. 28, 2010.

Gillianne Tedder | Bloomberg | Getty Images

The Iowa Public Employees Retirement System is terminating its relationship with Fisher Investments, pulling $386 million from the asset manager.

The Iowa plan, which holds $34 billion in total assets, announced its move on Friday and attributed its decision to sexist comments Ken Fisher, the billionaire founder of the firm, made at an investment conference last week.

The losses in pension assets for Fisher Investments is now about $1.3 billion.

"IPERS staff has taken time to evaluate this situation, and it is our opinion that Mr. Fisher's comments have damaged the credibility of the firm and its leadership," said Shawna Lode, a spokeswoman for the plan in a statement.

"As a result, the risk to IPERS is that the firm could lose investment talent, and/or it may be unable to recruit high caliber talent in the future," she said.

"Furthermore, the negative publicity will probably continue to be a major distraction to Fisher Investment personnel," Lode said.

The plan is weighing its transition options.

Earlier this week, Boston announced it would pull $248 million in pension assets from Fisher.

Similarly, the state of Michigan said it would withdraw $600 million of its pension fund assets. Philadelphia's board of pensions also yanked $54 million from Fisher.

Ken Fisher has since apologized for the comments.

"Some of the words and phrases I used during a recent conference to make certain points were clearly wrong and I shouldn't have made them," he said in a statement. "I realize this kind of language has no place in our company or industry. I sincerely apologize."

Controversial comments

CNBC obtained an audio recording last week of Fisher's comments at the Tiburon CEO Summit, as well as audio of him speaking at a previous conference.

Clips from both were featured on CNBC Power Lunch. Combined, they show that the money manager made flippant remarks about sex.

In the audio obtained by CNBC, Fisher says at the Tiburon conference: "Money, sex, those are the two most private things for most people," so when trying to win new clients you need to be careful.

He says: "It's like going up to a girl in a bar … [inaudible] …going up to a woman in a bar and saying, hey, I want to talk about what's in your pants."

Further, when Fisher was a speaker at the Evidence-Based Investing conference in 2018 he compared marketing mutual funds to propositioning a woman for sex at a bar.

"I mean the, the most stupid thing you can do, which is what every mutual fund firm in the world always did, was to brag about performance, uh, in, in a direct mail piece, which is a little bit like walking into a bar if you're a single guy and you want to get laid and walking up to some girl and saying, 'Hey, you want to have sex?'" Fisher said, according to audio obtained by CNBC.

Organizers of both conferences subsequently banned him from speaking again in the future.