A summer 2014 signing for The Tudors, Asafu-Adjaye started his career at Luton Town as a scholar. Unable to break into the Luton side at the beginning of the 2007–08 season, he was instead loaned to Conference Premier side Salisbury City in January 2008, in which time he earned several man-of-the-match accolades.

With Luton Town in administration and heading for relegation to League Two, new manager Mick Harford decided to recall Asafu-Adjaye from his loan spell in late March, and he made his Luton Town debut in a 2–1 home victory over Crewe Alexandra just days after returning to Kenilworth Road. In the 2008–09 season, Asafu-Adjaye established himself as the first choice right-back in the Luton team, and also started at centre-back in Luton's 3–2 victory over Scunthorpe United in the Football League Trophy final at Wembley Stadium.

On 30 April 2012 it was announced that Asafu-Adjaye was to be released from Luton. Asafu-Adjaye signed for Forest Green Rovers on a one-year contract; after an impressive opening few months at The New Lawn, he signed a contract extension that would keep him at the club until June 2014. On 28 April 2014, having made 40 league appearances in two seasons for the club, he was released.