His childhood name was Odaimaru (???). He held the title, passed down from his father, of Chinjufu-shogun, Commander-in-chief of the Defense of the North.

Yoriyoshi accompanied his father Minamoto no Yorinobu on his own missions to defend the Empire, quelling rebellions and disturbances. Thus he gained much of his knowledge of tactics and strategy. He fought in the Zenkunen War for twelve years starting in 1051, including the Battle of Kawasaki.