If the statement does indeed indicate FRAIS d'avis tiers détenteur, it means that the bank has charged you fees for the processing of an "avis tiers détenteur". So, at the very least, if you have had no such "avis tiers détenteur" from the tax office, the "hotel des impôts", or the local "Trésor Public", then the bank has withdrawn that money illegally. It is the bank's duty to investigate and explain why this money was taken from your account, and the benefit of the doubt is supposed to lie with the customer in such cases (i.e. unusual or unjustified activity on the account).
These "frais" are bank charges that are generated automatically when the bank receives notification of seizure of funds from a state tax organisation for non payment of a tax of any kind - water rates, rubbish disposal, childcare (if kids at school canteen in primary school), income tax, land tax, poll tax, speeding fines, etc. So, either the state has seized monies from your account, in which case, the bank charges are applicable, or it has not, in which case, the bank should not have debited those fees. Either way, the bank must be able to show why those fees were debited. For the state to emit an "avis tiers détenteur", it usually takes more than one reminder, and a fairly long interval between the first and subsequent reminders and actual notification of the avis tiers détenteur, so it seems unlikely that you would not have known about a due payment at some point, unless you throw your French mail in the bin, or unless they have targeted the wrong person (which happens).
The tax office should be able to provide a statement indicating that there are no known, current, or previous, payment incidents with them and that no avis tiers détenteur has ever been issued against you. You could then take this to the bank and get your unduly charged fees refunded.
As you have noticed, the state, under the guise of the DGFIP, does not need you to communicate your bank details to them, it can find all that out for itself if needs be, and it does, for anyone that gets caught in their sights (rightly or wrongly, as it happens).