Here's why:
* Preposition: "without" is the preposition.
* Object of the preposition: "me" is the object of the preposition, telling us what is being omitted.
The prepositional phrase functions as an adverbial phrase, modifying the verb "going" by explaining how the action is being performed.