Ahhh, I suspect it doesn't take over if you are the type of person that weeds it, makes sure it stays in bounds and regularly digs it up and redistributes it each year, as you do. You are apparently a very good and attentive gardener.
I, on the other hand, will not do that. If it is not right under my feet, (and even if it is) within a year or three, I will forget to do the necessary things to keep it contained and then I will have a yard full of horseradish.
Trust me on this one, I've lived with me all my life and know of which I speak.
I really shouldn't even start anything with the words "possibly invasive" used to describe it, but what's life without the occasional risk!