I do know there is documentation for the IB TWS Java API. However, instead of Java, I would like to establish a connection between Mathematica and the IBrokers API module using RLink as a 'proxy' to perform automated trading. How might I go about doing this?
It may be possible - as noted by @MarcoB its quite a convoluted way of doing things. I would just download R and RStudio and work from there.
However if you are set on using IBrokers via the R api via RLink try this (disclaimer it crashed for me on 10.1) :
Needs["RLink`"]; InstallR; REvaluate["install.packages(\"IBrokers\")"]; REvaluate["library(IBrokers)"];
Then follow the R API guide here. Note that it says the API is single threaded so you may run into some timing issues or other limitations if you are trying to execute or monitor multiple things at once.