It's a constructed language created specifically for the film by Paul Frommer. He designed it to be a complex and expressive language, incorporating elements of real-world languages like Polynesian, Native American, and African languages.
While the Na'vi language is not real in the sense that it's not spoken by any existing culture, Frommer created a rich vocabulary, grammar, and even a writing system for it. He also developed a comprehensive Na'vi lexicon and dictionary.