Taxe d'Habitation & deposit

Hi. We rented out our house in France (long-term rental) last year. Rental has now come to the end, the house is in perfect condition so we have to return the deposit. Due to the negligence of the estate agency, the taxe d’habitation was not paid by the renter and the money due was automatically taken out of my account (plus a small fine). The agency and the client are trying to deal with local tax office to rectify the problem but so far the taxe d’habitation was paid by me and that money is owed to me by the client. Do I have to return the deposit or can I hold onto it until the taxe d’habitation issue has been resolved? Many thanks. I would really appreciate any comments as the estate agency are useless. It is so frustrating.