It refers to the state of being merry, or a feeling of gaiety and joy.
Here are some examples:
* The party was filled with merriment. (Noun, the subject of the sentence)
* There was much merriment at the festival. (Noun, object of the preposition)
* Her laughter was a sign of her merriment. (Noun, object of the preposition)
You might be thinking of the adjective form, "merry".