If you're looking for something to wear instead of a sweater:
* Shirt (more generally)
* T-shirt (for warm weather)
* Tank top (for very warm weather)
* Dress (for women)
* Coat (for colder weather)
* Jacket (for colder weather)
If you're thinking of the material:
* Cloth (a broader term for fabric)
* Fabric (a broader term for material)
If you're focusing on the function:
* Cooling garment (a garment designed to keep you cool)
* Bare skin (not wearing anything at all)
Ultimately, the best antonym depends on the specific situation you're considering.