Here is how I would do it:

quotePattern = { "\"" , "'", "" };
stringPattern = quotePattern ~~ __ ~~ "by" ~~ __ ~~ quotePattern;

StringMatchQ[ #, stringPattern ] & /@ { "x by y", "\"x by y\"", "'x by y'" }

{ True, True, True }

Here is how I would do it:

quotePattern = { "\"" , "'", "" };
stringPattern = quotePattern ~~ __ ~~ "by" ~~ __ ~~ quotePattern;

StringMatchQ[ #, stringPattern ] & /@ { "x by y", "\"x by y\"", "'x by y'" }

{ True, True, True }

Edit

Be careful what you wish for ... as they say: The OP wants to extract "title by author" from a list of strings I assume. For example:

string = "Some surrounding text. \"The Machine That Changed the World\" by Womack, Jones, and Roos. And it fails.";

Now the above pattern will of course not work. We have to be more precise:

quotePatternMod = { "\"", "'" };
StringCases[ string,
    Shortest[a : quotePatternMod ~~ b : __ ~~ c : "by" ~~ d : ___ ~~ "."] :> a ~~ b ~~ c ~~ d
] 

{""The Machine That Changed the World" by Womack, Jones, and Roos"}