Olsen began his career at Næstved BK. During his time in Næstved he also played for the first time in Denmark. He made his debut in July 1980 against the Soviet Union. After a trial at Arsenal in 1979 he decided to change only two years later abroad. Olsen went to Ajax Amsterdam.
While he played for Ajax, he was known for playing a penalty. When playing in the Dutch Eredivisie between Ajax and Helmond Sport on December 5, 1982 Johan Cruyff played a penalty shortly, whereupon Olsen played the ball back played the Dutchman, who then could easily defeat the perplexed keeper. In the Netherlands, Olsen got the nickname flea because of its size, jumping ability and agility. He won with Ajax twice the Dutch championship and a Dutch Cup. During his time in the Netherlands, he was called up for the Danish squad for the 1984 European Championship in France, where Denmark was eliminated in the semifinals. Olsen was used twice and got two yellow cards.
In 1984, the Dane to England to Manchester United. With Manchester United he could win the 1985 FA Cup. During his time in Manchester, the appeal falls within the Danish squad for the 1986 World Cup in Mexico and the 1988 European Championship in Germany. In the 1986 World Cup, he came up with the Danes to the knockout stages, where you failed against Spain. Olsen played four times and scored three goals. At Euro 1988, he did not play. Here eliminated from the Danes in the group stage.
In 1988 he returned for two games his parent club Næstved BK. In these two games, it even managed to score a goal. In 1989, Olsen to France to Bordeaux. After a season in Bordeaux he hung still two years at SM Caen, where he ended his career due to a serious injury.
- Dutch champion: 1982, 1983
- Dutch Cup winner: 1983
- English Cup Winners: 1985