While it might seem like the prefix is "ag-", it's actually "a-" which is a common prefix meaning "without" or "not".
The word "aghast" comes from the Old English word "agæstan," meaning "to frighten" or "to terrify." The "a-" prefix in this case gives the word a negative connotation, implying a state of being "without" composure or being "not" at ease.