It's likely derived from the erratic and often nocturnal behavior of bats, which may have been seen as unpredictable and even frightening in the past.
Examples:
* "I'm going batty trying to find my keys!"
* "After working all night, he was starting to go batty."
The phrase is usually used humorously, but it can also be used to express genuine concern for someone who is behaving erratically.