I was asked to provide an employment reference for someone we recently dismissed as they were incompetent and useless. I told the new employer exactly how bad he was!
A worker can sue an employer if a negligent reference is given. The employer’s duty is to be fair and reasonable and to take reasonable steps to ensure there is no inaccuracy within it. There is therefore no truth in the myth that you cannot give a bad reference and there is an obligation of care to the receiver of the reference to be truthful accurate and fair. On the other hand it is wise to restrict unfavourable comments to matters about which you have properly investigated and have reasonable grounds for believing to be correct.
if you have very good reason for believing that he was useless and incompetent and there is no exaggeration of the facts, then you will be safe even if the ex employee does manage to get hold of a copy of the reference. It might be wise however be slightly more measured in future.