Playing her last tournament match before the Rio Olympics, the 36-year-old Williams lost seven straight games before holding serve in the third game of the second set. After Williams rallied to tie the match, Keys found her serve in the third set, ending the match with her 12th ace.
The 21-year-old Keys won the Wimbledon tuneup event in Birmingham, England, this year for her second WTA Tour title, then reached the fourth round at Wimbledon. She'll face 16th-seeded Anastasia Pavlyuchenkova of Russia in the quarterfinals.
Williams beat Barbora Strycova -- the Czech player whom Keys topped in the Birmingham final -- on Wednesday in her first match since losing to Britain's Johanna Konta on Sunday in the Bank of the West Classic final. Against Keys, Williams struggled with her serve.
"Her first serve was definitely slower than normal," Keys said. "But it was funny, her second serve was a lot slower, but because of the court, it was bouncing a lot higher than normal. So while her first serve was a little easier to return, her second serve was really tough."
Pavlyuchenkova beat fourth-seeded Agnieszka Radwanska of Poland.
"It's going to be a tough match," Keys said about Pavlyuchenkova. "She's always tricky because she definitely fights till the end, and she's going to hit a lot of winners."
Pavlyuchenkova let two match points slip away while leading 5-4 in the second set and then lost the game on a double fault. She rebounded in the third set to extend her longest run at the hardcourt event.
"I think I'm still recovering from the second set," Pavlyuchenkova said. "Basically, I thought I had the match in my pocket and lost it, so I'm happy with the mental side."
Radwanska won in 2014 in Montreal.
Bouchard was penalized a point for racket abuse in the third set, when she struggled to hit the lines to the disappointment of what had been a festive center-court crowd at Uniprix Stadium.
"I played some high-quality tennis this week -- I can't forget those two matches -- but two matches is not a whole tournament," Bouchard said. "I need to learn how to keep it going and deal with the pressure. I just felt I maybe panicked a little bit, tried to finish the points too soon."
"I'm still improving to playing better tennis, being more aggressive and just going for it," Kerber said. "I think today I was a little bit too excited to going for it. This is sometimes, I think, the problem."
Kerber will face Russia's Daria Kasatkina, a 7-5, 6-3 winner over seventh-seeded Roberta Vinci of Italy.
Fifth-seeded Simona Halep of Romania, a finalist last year in Toronto, beat 14th-seeded Karolina Pliskova of the Czech Republic 6-3, 6-3. Pliskova, the WTA Tour ace leader, had only five in the match.
"I knew that it's going to be tough because I don't get rhythm from her," Halep said. "You never know what to expect for the next point, so there's a bit of tension. I just had to stay patient for every ball and to keep fighting because I knew that if I stay there for every ball, she can miss more than me."