Using the option Evaluated->True or wrapping the first argument of Plot with Evaluated gives a 100x speed-up:
ls1[t_] := Quantity[10, "Nanometers"]/t
Plot[ls1[Quantity[t, "Seconds"]]/Quantity[1, "SpeedOfLight"], {t, 0, 10}] // Timing
ls2[t_] := Quantity[10, "Nanometers"]/t
Plot[ls2[Quantity[t, "Seconds"]]/Quantity[1, "SpeedOfLight"], {t, 0, 10},
Evaluated -> True] // Timing
ls3[t_] := Quantity[10, "Nanometers"]/t
Plot[Evaluate[ls3[Quantity[t, "Seconds"]]/Quantity[1, "SpeedOfLight"]],
{t, 0, 10}] // Timing
Update: Better yet, define the function to be plotted outside Plot wrapped with Evaluate:
ls4[t_] := Evaluate[Quantity[10, "Nanometers"]/Quantity[t, "Seconds"]/
Quantity[1, "SpeedOfLight"]]
Plot[ls4[t], {t, 0, 10}] // Timing
