Formal:
* Bravo pour l'effort! (Bravo for the effort!)
* C'est bien essayé! (That's well tried!)
* Vous avez fait de votre mieux! (You did your best!)
Informal:
* Bien joué! (Well played!)
* Pas mal! (Not bad!)
* T'as essayé! (You tried!) - This is very informal and might be considered slang.
More specific:
* Tu as presque réussi! (You almost succeeded!)
* Tu as bien travaillé! (You worked hard!)
* Tu as fait de ton mieux! (You did your best!)
The best way to say "good try" in French really depends on the situation and the person you are talking to.