As Jane says, if you live and work in France you are obliged to be registered and paying social contributions in France, really from Day One. This will then entitle you to health cover and other social benefits in France.
Once you are no longer UK resident you’re no longer covered by the NHS, or entitled to pay NICs, except in particular circumstances which require HMRC’s explicit approval.
Best get it sorted asap because any period you’ve spent working here without being registered, will potentially become an issue for various reasons - firstly because any time during which you’re not correctly exercising your FoM as a worker is unlikely to be counted towards legal residence if and when you apply for a carte de séjour, secondly because you’re not entitled to health cover from either the UK or France while you’re in limbo between the two and that obviously isn’t a good situation to be in, and thirdly because working unregistered (ie “on the black”) can lead to problems with URSSAF and/or DIRECCTE, as well as the tax office; in fact the tax office tends to be very forgiving, but URSSAF and DIRECCTE less so.
As regards claiming business expenses in France, that would depend which business regime you choose, there are a number of options. You could go to your local Chambre de Commerce for business advice - if you’re in an area where there are a lot of Brits, your CdeC will likely have an English speaking advisor.
Welcome to the forum, by the way!
EDIT - re-reading your post, do you continue to return to the UK to carry out the work, and if so do you have a portable health document to confirm to the French authorities that the UK remains responsible for your social security?
If so your income tax would be between you and HMRC. HMRC publishes a number of guides on what expenses are allowable, or failing that, I suggest you find a UK accountant.