So, "He tried to bone up on his vocabulary before the test" means he tried to learn a lot of vocabulary words in a short amount of time to do well on the test.
It's important to note that "bone up on" is a slightly informal idiom, so it might not be appropriate for all situations.