Is door jam or jamb proper English?

"Jam" is a different word entirely, meaning a sticky, sweet food or a crowded situation.

So, the frame around a door is called a door jamb.