The 18-year-old stunned Gauff 6-1, 6-4 in just 62 minutes in the last 16 of the WTA 1000 tournament in Montreal.
Mboko broke the two-time Grand Slam champion three times as she raced through the opening set in only 26 minutes.
In the second set, Mboko saved four break points in the seventh game before striking the decisive blow when Gauff served to stay in the match at 4-5.
The Canadian wildcard was imperious on serve as she won 79% of her first serve points, 62% behind her second delivery and saved all five of the break points she faced.
Gauff hit 37 double faults across her first two matches in Canada and, while her tally of six double faults in this match was less alarming, she was hamstrung by her serve once again.
The world No 2 landed just 43% of her first serves and won only 44% (12 of 27) of the points played on her second serve. The 21-year-old American hit 23 unforced errors as she struggled on her forehand.
On the crowd support she is receiving at her home event, Mboko said: “Every day they’ve been coming to all my matches and supporting me a lot.
“I couldn’t be more grateful. I’ve never really had so many people cheering for me in a tournament, especially a WTA tournament.
“This is kind of the only tournament Canada has, WTA tournament. To have that kind of support at a tournament like this, it’s amazing. I couldn’t be more grateful, and I think it’s so cool that so many Canadians get to come and support me.”
Asked what she was thinking as Gauff frequently committed unforced errors, Mboko said: “Well, I mean, nothing really crosses my mind just because, I mean, that’s the game. People make errors. It happens. I make errors too.
“So I was just really focused on myself and what I had to do. I didn’t really focus as much as what she was doing in the match because I just wanted to play as solid of a game as I possibly could.
“I just wanted to do what I was supposed to do in the match and hope for the best.”
Mboko started the tournament as the world No 85 and she has surged 30 places to a projected new career-high of 55th in the Live WTA Rankings with her run to the quarterfinals.
The teenager will face world No 51 Jessica Bouzas Maneiro in the last eight, and victory would likely see her break into the top 50.