Heat oil in a large pot. Use enough oil that there’s about 2 ½ inches-3 inches in the pot. Oil should be 350°F with an instant read thermometer.
In a small bowl, whisk the milk, eggs, butter, and vanilla. Set aside.
In a large bowl whisk the flour, brown sugar, baking powder, salt, cinnamon, ginger, and nutmeg. Whisk the wet ingredients into the dry ingredients.
Using a rubber spatula, fold in the chopped peaches. Don’t overmix.
Using a ¼ cup or a large cookie scoop, dollop the batter into the hot oil, cooking 2-3 at a time. Cook the fritters, turning them over so both sides cook, about 2-3 minutes per side for a total of about 4-5 minutes. Fritters should be a deep golden brown.
Use a slotted spoon to scoop the fritters out of the oil. Transfer the fritters to a paper towel lined plate.
Let the fritters cool slightly, then dip in the glaze so that they are coated on all sides. Set the glazed fritters on a cooling rack. Let the glaze set and then serve. Best if eaten the same day they are made.