Allardyce, 68, was parachuted in to rescue the club in May with only four games remaining but Leeds finished second from bottom, five points adrift of the safety zone and six above Southampton.
He lost three and drew one of his matches in charge after replacing Javi Gracia.
Leeds said an announcement on the former England manager's replacement would be made in the coming weeks.
"It has been an honour to manage Leeds United, a great club with an incredible fan base, who deserve to be in the Premier League," Allardyce said in a statement on the club website.
"At this stage in my career I am not sure taking on this challenge, which is potentially a long-term project, is something I could commit to," he added.
"But I wish the club every success for the future and hope the club returns to the Premier League, where they belong."
Leeds chief executive Angus Kinnear thanked Allardyce "for being brave enough to step in and do all he could to save us."