With little help and algebraic manipulation we can solve:
eq1 = F5'[x] - F4[x] == 0;
eq2 = F1'[x] - a*x*F4[x] == 0;
eq3 = F4'[x] + F1[x] + a*x*F5[x] == 0;
EQ = D[eq3, x] /. F1'[x] -> a*x*F4[x] /. F4''[x] -> F5'''[x] /. F4[x] -> F5'[x]
f5 = F5[x] /. First@DSolve[EQ, F5[x], x][[1]] // Simplify(*Solution for F5[x]*)
f4 = D[f5, x] // Simplify(*Solution for F4[x]*)
f1 = (F1[x] /. First@Solve[eq3, F1[x]]) /. (F5[x] -> f5) /. F4'[x] -> D[f4, x] // Simplify(*Solution for F1[x]*)
And the end we can check if solutions are TRUE
{eq1 /. F5'[x] -> D[f5, x] /. F4[x] -> f4,
eq2 /. F1'[x] -> D[f1, x] /. F4[x] -> f4,
eq3 /. F4'[x] -> D[f4, x] /. F1[x] -> f1 /. F5[x] -> f5} // Simplify
{True, True, True}
UPDATE: 16.05.2021
Mathematica 12.3
can solve. I tried on Wolfram Cloud.
DSolve[{F5'[x] == F4[x], F1'[x] == a x F4[x], F4'[x] + F1[x] + a x F5[x] == 0}, {F1[x], F4[x], F5[x]}, x]
{{F1[x] -> C[1]*HypergeometricPFQ[{-1/6}, {-1/3, 1/3}, (-2*a*x^3)/9] +
(2^(2/3)*a^(2/3)*x^2*C[2]*HypergeometricPFQ[{1/2}, {1/3, 5/3},
(-2*a*x^3)/9])/(3*3^(1/3)) +
(2*2^(1/3)*a^(4/3)*x^4*C[3]*HypergeometricPFQ[{7/6}, {5/3, 7/3},
(-2*a*x^3)/9])/(9*3^(2/3)),
F4[x] -> -(x*C[1]*HypergeometricPFQ[{5/6}, {2/3, 4/3}, (-2*a*x^3)/9]) +
(C[2]*((2*2^(2/3)*a^(2/3)*x*HypergeometricPFQ[{1/2}, {1/3, 5/3},
(-2*a*x^3)/9])/(3*3^(1/3)) -
(2^(2/3)*a^(5/3)*x^4*HypergeometricPFQ[{3/2}, {4/3, 8/3},
(-2*a*x^3)/9])/(5*3^(1/3))))/(a*x) +
(C[3]*((8*2^(1/3)*a^(4/3)*x^3*HypergeometricPFQ[{7/6}, {5/3, 7/3},
(-2*a*x^3)/9])/(9*3^(2/3)) -
(2*2^(1/3)*a^(7/3)*x^6*HypergeometricPFQ[{13/6}, {8/3, 10/3},
(-2*a*x^3)/9])/(45*3^(2/3))))/
(a*x), F5[x] -> C[1]*(-(HypergeometricPFQ[{-1/6}, {-1/3, 1/3},
(-2*a*x^3)/9]/(a*x)) +
HypergeometricPFQ[{5/6}, {2/3, 4/3}, (-2*a*x^3)/9]/(a*x) -
(5*x^2*HypergeometricPFQ[{11/6}, {5/3, 7/3}, (-2*a*x^3)/9])/8) +
C[2]*(-1/3*(2^(2/3)*x*HypergeometricPFQ[{1/2}, {1/3, 5/3},
(-2*a*x^3)/9])/(3^(1/3)*a^(1/3)) +
(2^(2/3)*x*HypergeometricPFQ[{3/2}, {4/3, 8/3},
(-2*a*x^3)/9])/(3^(1/3)*a^(1/3)) -
(3*3^(2/3)*a^(2/3)*x^4*HypergeometricPFQ[{5/2}, {7/3, 11/3},
(-2*a*x^3)/9])/(80*2^(1/3))) +
C[3]*((-16*2^(1/3)*HypergeometricPFQ[{7/6}, {5/3, 7/3},
(-2*a*x^3)/9])/(9*3^(2/3)*a^(2/3)) -
(2*2^(1/3)*a^(1/3)*x^3*HypergeometricPFQ[{7/6}, {5/3, 7/3},
(-2*a*x^3)/9])/(9*3^(2/3)) +
(2*2^(1/3)*a^(1/3)*x^3*HypergeometricPFQ[{13/6}, {8/3, 10/3},
(-2*a*x^3)/9])/(5*3^(2/3)) -
(13*a^(4/3)*x^6*HypergeometricPFQ[{19/6}, {11/3, 13/3},
(-2*a*x^3)/9])/(900*6^(2/3)))}}