'had to' alone means 'an action that you did in the past (an action you believed to be the right choice)
I had to do it I left my keys so I had to return home I had to tell him about it.
Your example however expressed a different meaning. It expresses anger or disapproval. Like, "My dream of working in Japan would have finally become a reality, but the pandemic had to destroy it. -.-'' (this is actually true and I'm very frustrated about it)