I confirm the problem occurs with V11.0.1 running OS X. It looks a bug to me. Here is a work-around.
GeoListPlot[
{Entity["Country", "Algeria"], Entity["Country", "Egypt"],
Entity["Country", "Morocco"], Entity["Country", "Madagascar"],
Entity["Country", "Tunisia"], Entity["Country", "Reunion"],
Entity["Country", "SouthAfrica"], Entity["Country", "Ethiopia"],
Entity["Country", "Zimbabwe"]},
GeoLabels -> (Text[#2[[2]], Reverse[#3[[1]]]] &)]

Edit
It occurred to me, after sleeping on it, that some viewers of this answer might be interested in how I worked out a work-around, so I present a description of the process.
The documentation articleGeoLabels
says that the option wants a function with three arguments, which are
- A graphic representing of the geographic object (Polygon) or its location (Point).
- the region entity itself
- the region entity's position on the map (latitude, longitude)
The article also shows an example of such a function where the label appears as a tooltip. I adapted that to a reduced version of the problem as a tool to examine the arguments as shown below.
GeoListPlot[
{Entity["Country", "Madagascar"], Entity["Country", "Reunion"],
Entity["Country", "SouthAfrica"], Entity["Country", "Zimbabwe"]},
GeoLabels -> (Tooltip[#1, {#1, #2, #3}] &)]

Because the 2nd argument is the region entity and the 2nd argument of that is the region's name, I use #2[[2]]
to extract the name. Also, because the the position is given as {latitude, longitude} but plotted as {longitude, latitude}, I reverse the position extracted from the 3rd argument. I also add 0.75 degrees of latitude so that Reunion is not hidden by its label.
GeoListPlot[
{Entity["Country", "Madagascar"], Entity["Country", "Reunion"],
Entity["Country", "SouthAfrica"], Entity["Country", "Zimbabwe"]},
GeoLabels -> (Text[#2[[2]], Reverse[#3[[1]]] + {0, .75}] &)]
This produces the desired labeling.

Update
I reported the OP's problem to Wolfram tech support. I have received an answer confirming it is a bug in V11. I quote the relevant part of the reply.
Your observation is correct. The simple expression [GeoLabels -> True] should work. The issue has been reported to our developers and it will hopefully be fixed in the future release of Mathematica.
10.4.1.0
. What version are you in? $\endgroup$