Reflecting on this he warned the regulator had to be careful not to stop a genuine person who was just caught by unforeseen circumstances from earning their living.
He even believes there could be grounds to take the regulator to court should that happen.
Redistributing fines
Richards said the main source of claims and FSCS costs were the spikes in business that could be observed in recent years, for instance with DB transfer advice.
"It was inevitable that things like DB transfers were going to be under scrutiny because it was a spike. So it was inevitable that we were going to end up where we've ended up," he said.
The real answer to the question of rising levies, he said, was to "now re-consider reinstating the reallocation of funds back into the system that rewards the good players and acts as a form of dividend.
"It will also provide more funding, so it will take away the uncertainty for firms, the additional impact of unwelcoming levies, but it also provides additional funding for financial education and engagement."
He said low savings rates were one of the prime issues in this country, especially in the cost of living crisis which could lead to more people borrowing themselves out of financial hardship.
Research has shown, he added, that one of the main reasons people weren't seeking financial advice was a lack of awareness of why they should, rather than a lack of trust in the profession.
The oft-cited cost also was not the main reason, more "not understanding what you get for the cost of advice", said Richards.
There was a lot more basic education and engagement in the past because it was easy to access via bank branches and insurance agents.
"The reason we are calling on the government to start investing some of the fines, rather than keep taking it as a stealth tax or an indirect tax to start reinvesting back into financial services and consumer education, is because we've had that lack of engagement."
Meanwhile, Richards said he was hopeful the government's review of the advice guidance boundary will lead to some positive development in the sector when it comes to providing guidance.
"I think we will see a more pragmatic approach for a number of reasons. Politicians tend to be good at looking at strategy, they're not regulators.
"They're looking at what the problem is and then they're looking at the most logical and pragmatic solution."
He thinks the solution will come from big manufacturers, who will not give advice but guidance that helps people decide whether they want to buy a product.
