I just upgraded to Mathematica 10.4 (Site Home Use) on Ubuntu 15.10 (x86-64). After entering the activation key and clicking Activate, I got a message "Activation was successful", and then the window disappeared and nothing happened.
ps showed that Mathematica was still running, with a number of subprocesses, some of which were
strace showed that both were in an apparently infinite loop of
futex calls. Lacking any better ideas, I killed one of them (
kill -TERM didn't successfully kill it, so I used
kill -9). A couple seconds later, I was presented with the license agreement screen, and was able to start using Mathematica.
Ever since then, trying to start it (running
Mathematica from the command line) results in similar behavior: processes start but no windows appear. Killing one of the
WolframKernel processes (with
kill -9) causes the following window to appear:
I can then click "New Notebook" and Mathematica seems to work normally.
Maybe this is a question for Wolfram Support, but I thought I would first see whether it is a well-known problem, or if there's something obvious I should try.
Update. With Mathematica 11 the problem seems to be fixed.