If you're thinking of amusement as a feeling of enjoyment:
* Boredom
* Displeasure
* Disinterest
* Tedium
* Dullness
If you're thinking of amusement as an activity or entertainment:
* Work
* Duty
* Seriousness
* Responsibility
If you're thinking of amusement as a way to laugh or find something funny:
* Seriousness
* Solemnity
* Gravity
The best antonym will depend on the specific situation and what you're trying to convey.