Here are some ways to understand it:
* Not relevant: The information or point being made doesn't have a bearing on the current discussion.
* Unimportant: It doesn't matter in the grand scheme of things.
* Off-topic: The subject is a tangent and doesn't contribute to the central issue.
Example:
"You mentioned that the restaurant was noisy, but that's neither here nor there. The food was delicious, and that's what matters."
In this case, the noise level is irrelevant to the main point, which is the quality of the food.