I have fought with this problem for a while now and have finally solved it. I tried connecting using windows authentication first as per this post but I just couldn't get it to work. I also tried to use the GUI (wizard) to create the connection but this seemed buggy. The details below refer to a database on a server which has the Mathematica network licence manager on it as well as Mathematica 8 installed on it. I use Mathematica 8 on my local machine with one of the licences from the network licence.
This is how i got it to work:
I made sure that I could connect via SQL Server Management Studio to the database on the server from my local machine using the SQL Authentication credentials I had. I could connect thus I knew there were no network problems and my SQL Authentication credentials were working.
I found the correct name of the server - it was the name as used when connecting via SQL Server Management studio but without the SQL instance included.

I selected two databases that I wanted to connect to - they are called master and model
I compiled a notebook as below and was able to connect.
Loading the package
Needs["DatabaseLink`"]
Details:
ECSDBN-SRV-09.enerserv.local
sa
Password
Catalog: master
192.168.1.11
Open the connection
conn = OpenSQLConnection[
JDBC["Microsoft SQL Server(jTDS)", "ECSDBN-SRV-09/model"],
"Username" -> "sa", "Password" -> "N3tC0nf1g"]
SQLConnection[1, "Open", "Catalog" -> "model",
"TransactionIsolationLevel" -> "ReadCommitted"]
conn1 = OpenSQLConnection[
JDBC["Microsoft SQL Server(jTDS)", "ECSDBN-SRV-09/master"],
"Username" -> "sa", "Password" -> "N3tC0nf1g"]
SQLConnection[2, "Open", "Catalog" -> "master",
"TransactionIsolationLevel" -> "ReadCommitted"]
(*Here I have not specified the database nane after the server name and it just seems to pick the one on the top of the database list.*)
conn2 = OpenSQLConnection[JDBC["Microsoft SQL Server(jTDS)", "ECSDBN-SRV-09"],
"Username" -> "sa", "Password" -> "N3tC0nf1g"]
SQLConnection[3, "Open", "Catalog" -> "master",
"TransactionIsolationLevel" -> "ReadCommitted"]
(*Look at the open connection to make sure.*)
SQLConnections[]
{SQLConnection[1, "Open", "Catalog" -> "model",
"TransactionIsolationLevel" -> "ReadCommitted"],
SQLConnection[2, "Open", "Catalog" -> "master",
"TransactionIsolationLevel" -> "ReadCommitted"],
SQLConnection[3, "Open", "Catalog" -> "master",
"TransactionIsolationLevel" -> "ReadCommitted"]}
(*Check what tables exist through the current connection*)
SQLTables[conn]
{}
SQLTables[conn1]
{SQLTable["MSreplication_options", "TableType" -> "TABLE"],
SQLTable["spt_fallback_db", "TableType" -> "TABLE"],
SQLTable["spt_fallback_dev", "TableType" -> "TABLE"],
SQLTable["spt_fallback_usg", "TableType" -> "TABLE"],
SQLTable["spt_monitor", "TableType" -> "TABLE"],
SQLTable["spt_values", "TableType" -> "TABLE"]}
SQLTables[conn2]
{SQLTable["MSreplication_options", "TableType" -> "TABLE"],
SQLTable["spt_fallback_db", "TableType" -> "TABLE"],
SQLTable["spt_fallback_dev", "TableType" -> "TABLE"],
SQLTable["spt_fallback_usg", "TableType" -> "TABLE"],
SQLTable["spt_monitor", "TableType" -> "TABLE"],
SQLTable["spt_values", "TableType" -> "TABLE"]}
(*Check that I can access data from one of the tables in the connection using the SQL query approach*)
SQLExecute[conn1, "SELECT * FROM MSreplication_options",
"GetAsStrings" -> True] // InputForm
{{"transactional", "1", "90", "0", "0", "0"}, {"merge", "1", "90", "0", "0", "0"}, {"security_model", "1", "90", "0", "0", "0"}}
(*Try another server*)
conn = OpenSQLConnection[
JDBC["Microsoft SQL Server(jTDS)", "ECSDBN-SRV-04.enerserv.local"],
"Username" -> "sa", "Password" -> "P@ssw0rd"]
JDBC::error: Network error IOException: Connection refused: connect >>
$Failed
However when I tried the same thing with another server which does not have Mathematica installed on it (this may or may not be the problem) I could not connect. Also, the server I was able to connect to has two seperate instances of SQL on it - SQL 2005 and SQL 2008. Mathematica connected me to the 2005 instance by default and I can't find out how to connect to the 2008 instnace. I hope this helps a little bit.
Reference material: DatabaseLink/tutorial/Overview