If I am interpreting the question correctly, you are trying to remove the pivot pin holding in the peg, and the pin will only slide 80% of the way out before hitting the bottom of the rear brake lever?
I think
@Rashid510 advice above is correct. If you remove the two bolts he highlighted in yellow, you can swing the rear brake lever out of the way without completely disconnecting it.
I didn't even think about this issue since I swapped out the rear brake lever for a left-hand rear brake on the bars. So it wasn't an issue when I swapped my pegs, since no brake lever in the way.