It wasn't just mail these connections were appearing on which is why the connection limit is affecting everything.

It appears to be some kind of TCP SYN attack as the connections are in the SYN state as though the server has responded to the SYN request with a SYN+ACK and is waiting for the client to return ACK.