Perhaps a read of this dossier may change your view in part
www.lefourquet.net/Hardup-dossier.doc
and also
and read clause 4.2 of that document.
4.2 This would include the status and entitlements of the approximately 2 million UK citizens living, working and travelling in the other 27 Member States of the EU. They all currently enjoy a range of specific rights to live, to work and access to pensions, health care and public services that are only guaranteed because of EU law. There would be no requirement under EU law for these rights to be maintained if the UK left the EU. Should an agreement be reached to maintain these rights, the expectation must be that this would have to be reciprocated for EU citizens in the UK.
Without the overall protection of the EU regulations ('laws') The family X would become foreigners in France as foreign as Chinese, Americans, Peruvians. At the moment they are European Citizens. Without that EU protection they would be subject to any changes that either the UK could inflict on their finances from the UK , or the French on them as étrangers - without any redress.