Well I think I figured it out, I can rule out the soil being the problem: I'm growing lots of other herbs in the same stuff and those taste just fine with no hint of fish. I looked borage up regarding nutritional value and noticed they're known to have a high amount of GLA. The same stuff is in flax seed oil, which I've had in the past and hated due to the fishy taste. I guess GLA and my taste buds just don't like each other
I'll be on the lookout for salad burnet seeds. In the meantime I'll keep the borage for the flowers and hope they attract bees.