While Wilde did love puns and was known for his wit, there's no documented source for this particular quote. It's likely a popular saying that became associated with him over time.
It's also worth noting that many people have a fondness for puns, even if they may not be considered highbrow humor!