IMO it's the price you pay for starting early, before the summer temperatures have kicked in - but depending on the variety, they may grow those little fruit nubs later, or not. For example, every year Black Cherry would set these 'empty nests' and never grow them - all the action happened further up the plant. But Anna Russian and other hearts or bigger fruited varieties I have seen set the little nubs and grow them later on, after the temperature has gone up, or after the first fruit have been harvested. Black Sea Man would set a full first cluster but only grow some of them to the point of ripeness, then afterwards the little ones would come on.
The only thing to watch for, if those little nubs are black or turn black they're toast.
Otherwise, leave em and hope for the best.