Yep. Every time you cross back to HM you are stacking up the recessives for dwarf size and small fruit, making it more difficult to find what you're seeking.
You should be able to find something promising in the F2, but if you don't, I would suggest trying a different parent for "full size tomato" instead of trying to force it by backcrossing. Why a different parent? Because IMO taste genetics are extremely diverse and complex (many many genes involved), and if results are not as expected, there may be issues such as linkage and epistasis affecting this parent combination. You can also try crossing the F1 or F2 to a third parent, as a workaround for epistasis. Or even better, you could start by crossing HM to two different 'large fruit' parents. Then you have two shots at the HM flavor genes by crossing the F2's.
As KarenO pointed out, each time you make a new cross you are back at the F1 stage and seven generations from stable. But honestly you are better off to make the new cross one way or another, if you're not finding that key trait that you wanted to carry forward. Or alternatively, just keep growing out more and more F2's until you find it.
Large size is one of the harder traits to recover, btw. So do expect to grow lots of F2's to find a big one with the taste you desired.
