Here's why:
* "Challenging" is a present participle adjective, describing the nature of the teaching style.
* "Challenges" is a verb in the third person singular form, implying an action performed by the teaching style.
Since we're describing the *effect* of the teaching style, the adjective form "challenging" is the better fit.
Therefore, the complete sentence should be:
Their innovative teaching style encourages creativity while challenging students to think critically and...