This should do the trick (for example, if you would like to use a semicolon as a field separator):
dataset = Table[i^2 - 10.*j, {i, 10}, {j, 10}];
Export["dataset.csv", ToString[ScientificForm[datasetdataset/.x_?NumberQ:>ToString[ScientificForm[x, 10, NumberFormat -> (Row[{#1, "E"If[#3 == "", #3, "E"], #3}] &)]], "Table", "FieldSeparators" -> ";"]
Depending on the structure of your dataset, you might need to apply ScientificForm to its members separately (or only some of them, or only numeric ones).
The advantage of "Table" over "CSV" is that the separator is not fixed to comma, but can be changed. See also the documentation: