I'm not sure, if it is possible to do that directly in Mathematica, but there is a command line tool called youtube-dl. It allows the direct extraction of the audio by using the -x option, see here Post-processing Options.
Example:
youtube-dl https://www.youtube.com/watch?v=jNQXAC9IVRw -x --audio-format mp3
You can use Run[] to run the command directly from Mathematica.