The correct way to say it is:
* He was continuing to do his homework.
Here's why:
* Continuing is a verb, and verbs need a direct object or an infinitive phrase to make sense.
* To do is the infinitive phrase that acts as the object of the verb "continuing."