So, "he tried to bone up his vocabulary before the test" means that he tried to learn and memorize a lot of vocabulary words in a short amount of time before the test.
It's important to note that "bone up" is informal and can sound a bit crude. In formal settings, you might say "he tried to brush up on his vocabulary" or "he tried to review his vocabulary" instead.