If "finally" implies a long wait:
* Initially
* At first
* Previously
* Before
If "finally" implies an end to something:
* Initially
* At the beginning
* Previously
* Before
If "finally" implies a change of state:
* Initially
* At first
* Previously
* Before
If "finally" implies a sense of relief:
* Unhappily
* Sadly
* Regrettably
* Unfortunately
The best opposite will depend on the specific context and the intended meaning.