What word would be correct for Her shirt cuffs were fray?

The correct word to use is frayed.

The sentence should read: Her shirt cuffs were frayed.

"Frayed" means worn or damaged at the edges, which is the appropriate description for the condition of shirt cuffs.