Chelsea manager Thomas Tuchel now has a five-game unbeaten streak across all competitions since taking over for Frank Lampard.
Abraham gave Chelsea the game-winner in the 64th minute when he tapped in a low cross from Reece James.
Video replay appeared to show Abraham offside but the match did not allow for VAR as the match was held at a Championship venue.
Abraham also made a stellar play on the defensive side when he headed the ball over the bar after keeper Kepa Arrizabalaga had fumbled it toward Michael Sollbauer.
Chelsea will next face Sheffield United in the quarterfinals of the competition.