It describes a feeling of regret, sorrow, or remorse.
For example: "He gave a rueful smile at his own foolishness."