0
$\begingroup$

I am truly confused with this one, and maybe someone could help me to clear this up.

I am running a Table[] that runs multiple neural networks in series on OpenAI's open gym for reinforcement learning. In my program, when Table[] is configured with n as the iterator, like the following:

Table[evaluateSingleNetworkOnOnePole[population[[n]], session, numberFrames], {n, Length[population]}]

The output is the following, where the values are incorrect because they are far too low (there is some slight variation in output each time within a few integer values due to the nature of OpenAI's gym):

{9., 8., 9., 8., 9., 10., 9., 10., 9., 10., 8., 10., 10., 10., 9., \
9., 10., 10., 10., 9., 9., 9., 9., 9., 9., 10., 9., 9., 9., 9., 10., \
9., 10., 10., 11., 10., 8., 9., 9., 10., 10., 10., 9., 10., 8., 9., \
8., 10., 9., 10., 10., 8., 10., 10., 10., 9., 9., 10., 9., 8., 9., \
10., 10., 10., 10., 8., 10., 9., 10., 8., 9., 9., 9., 8., 9., 8., 9., \
9., 9., 10., 10., 10., 9., 9., 9., 10., 11., 8., 10., 9., 10., 9., \
10., 9., 9., 8., 9., 8., 8., 8., 10., 10., 9., 10., 10., 9., 10., 8., \
9., 10., 9., 9., 10., 9., 9., 10., 10., 9., 10., 10., 10., 8., 9., \
8., 9., 9., 9., 10., 10., 8., 10., 9., 9., 9., 8., 9., 10., 10., 9., \
10., 9., 9., 9., 10., 8., 8., 8., 9., 9., 8., 9., 8., 10., 10., 9., \
10., 10., 9., 10., 9., 10., 10., 9., 10., 10., 10., 9., 10., 9., 10., \
9., 9., 8., 10., 10., 9., 9., 9., 9., 9., 11., 10., 9., 9., 9., 10., \
9., 10., 9., 9., 9., 10., 10., 9., 9., 10., 10., 9., 10., 10.}

Where none of the values ever are over 11. However, if I change the iterator to x, then the code works as I originally expected:

Table[evaluateSingleNetworkOnOnePole[population[[x]], session, numberFrames], {x, Length[population]}]

Giving:

{48., 13., 10., 100., 59., 44., 10., 9., 9., 12., 9., 13., 14., 36., \
100., 10., 10., 10., 46., 42., 8., 11., 9., 10., 10., 9., 25., 30., \
33., 81., 18., 11., 100., 9., 32., 29., 14., 9., 11., 58., 8., 13., \
15., 9., 9., 9., 60., 83., 100., 10., 10., 9., 12., 22., 10., 14., \
10., 100., 14., 34., 9., 37., 36., 28., 22., 12., 26., 67., 12., 9., \
18., 70., 49., 10., 8., 10., 33., 16., 15., 11., 9., 13., 21., 40., \
23., 9., 20., 9., 11., 76., 36., 44., 9., 9., 30., 19., 41., 47., \
18., 11., 38., 41., 9., 55., 14., 10., 54., 16., 41., 76., 10., 56., \
10., 9., 11., 10., 11., 11., 13., 9., 10., 11., 8., 9., 20., 53., \
13., 9., 43., 10., 17., 29., 9., 10., 8., 10., 9., 27., 14., 15., \
29., 23., 8., 77., 12., 10., 56., 13., 10., 65., 38., 19., 27., 15., \
13., 16., 10., 9., 86., 100., 9., 12., 8., 10., 10., 38., 9., 36., \
49., 54., 11., 14., 17., 17., 37., 28., 39., 10., 20., 15., 10., 10., \
25., 58., 10., 9., 100., 48., 10., 31., 25., 56., 17., 9., 16., 50., \
15., 10., 10., 35.}

Does anyone know what may be causing this? As more background, n does not have any values saved to it. Performing Clear[n] makes no difference in the strange outputs. I do have some other functions that use n as an iterator, but they are within Module[]s and are just used as iterators in functions like Table[].

Any help on this would be very much appreciated, as I feel I must be missing something with this one.

EDIT: For those interested, here is all of the code pertaining to the error I am getting. I have, as mentioned, been able to solve the problem by just replacing n with x, but still do not know why n is not working as intended:

numberOfInputs = 4;
numberOfOutputs = 1;
populationSize = 200;
numberFrames = 500;
structure = {100, Tanh, 100, Tanh, numberOfOutputs, Tanh};
session = initializeCartpoleSession[];
population = (time = UnixTime[];
   Table[SeedRandom[time + x];
    NetInitialize@NetChain[structure, "Input" -> numberOfInputs], {x, 
     populationSize}]);

(*Initialize the Python session which will be used for OpenAI Gyma*)
initializeCartpoleSession[] := Module[{session},
  session = StartExternalSession["Python"];

  ExternalEvaluate[session,
   "import gym
   env = gym.make('CartPole-v0')
   env.reset()"];

  ExternalEvaluate[session,
   "def runCartPoleOnce(action, env):
      observation, reward, done, info = env.step(action)
      return observation, reward, done, info"];

  session
  ]

(*Send in an action of left or right (0 or 1) and get the reward and \
state output as a result*)
runCartPoleOnce[session_, action_] := Module[{},
  ExternalEvaluate[session, 
   StringJoin["runCartPoleOnce(", ToString[action], ",env)"]]
  ]

(*Take a network and run it through the simulation for numberFrames \
frames*)
evaluateSingleNetworkOnOnePole[network_, session_, numberFrames_] := 
 Module[{action, reward, input},
  (*Reset the state of the cartpole*)
  action = 
   Normal[network[ExternalEvaluate[session, "env.reset()"], 
       TargetDevice -> "GPU"]][[1]] /. n_Real -> If[n < 0., 0, 1];

  Total[Total[
    Table[
     action = 
      Normal[network[{input, reward} = 
           runCartPoleOnce[session, action][[;; 2]]; input, 
          TargetDevice -> "GPU"]][[1]] /. n_Real -> If[n < 0., 0, 1];
     reward
     , {numberFrames}]
    ]]
  ]

(*Evaluate all the networks in the population, returning their \
fitnesses*)
evaluateOnePopulation[population_, session_, numberFrames_] := 
 Module[{},
  Table[evaluateSingleNetworkOnOnePole[population[[n]], session, 
    numberFrames], {n, Length[population]}]
  ]

Even running on a different kernel, when I run:

evaluateOnePopulation[population[[;; 10]], session, numberFrames]

I get back:

{9., 8., 10., 9., 9., 9., 9., 9., 10., 10.}

However, when I switch n to x in:

(*Evaluate all the networks in the population, returning their \
fitnesses*)
evaluateOnePopulation[population_, session_, numberFrames_] := 
 Module[{},
  Table[evaluateSingleNetworkOnOnePole[population[[x]], session, 
    numberFrames], {x, Length[population]}]
  ]

And run the same command again, I get:

{45., 10., 8., 10., 10., 40., 10., 10., 59., 10.}

Which is the expected result, as the values are no longer limited to being 11 or lower. Could this be a bug, or is there something I am overlooking?

$\endgroup$
5
  • 5
    $\begingroup$ It is hard to know without seeing the definition of evaluateSingleNetworkOnOnePole -- I suggest you try to devise a minimal example that illustrates the problem. Perhaps it is the issue described in Do Table iteration variables need to be localized using Module? $\endgroup$ – WReach Jan 11 '20 at 7:15
  • 3
    $\begingroup$ No, n is does not have a buiIt-in meaning. I propose to restart the kernel with Exit and to to again. $\endgroup$ – Henrik Schumacher Jan 11 '20 at 8:26
  • 2
    $\begingroup$ As is, there are not a lot to go by; some observations/suggestions: 1. there are exactly 3 instances of 11 in the first list instead of none; the corresponding number in the second list is 11 2. the first list appears as a lower bound of some sort of the second list (using ListLinePlot helps) 3.1. if I'd have to guess, what you're dealing with is probably some kind of premature convergence 3.2. From the code, there doesn't seem to exist something that suggests n is somehow relevant to the problem. 4. Try using other iterator symbols and see what happens $\endgroup$ – yosimitsu kodanuri Jan 11 '20 at 9:38
  • $\begingroup$ I have edited the question to include the details of the program, so one can try and recreate the error. Sorry it is not more brief but, seeing as how I am not sure what is causing the error, this is the most compact version I could currently give. $\endgroup$ – Jmeeks29ig Jan 11 '20 at 18:04
  • $\begingroup$ @WReach - It turns out that your suggestion worked - I went through and localized n to the modules in which it was used and the program now works - thanks! I was not aware that Table[] does not localize variables automatically. $\endgroup$ – Jmeeks29ig Jan 11 '20 at 18:10
3
$\begingroup$

As suggested by WReach, I added n to the Module[]s local variables and that solved the problem:

(*Evaluate all the networks in the population, returning their \
fitnesses*)
evaluateOnePopulation[population_, session_, numberFrames_] := 
 Module[{n},
  Table[evaluateSingleNetworkOnOnePole[population[[n]], session, 
    numberFrames], {n, Length[population]}]
  ]

Now, when I evaluate:

evaluateOnePopulation[population[[;; 10]], session, numberFrames]

I get back the intended results:

{9., 9., 9., 76., 60., 10., 10., 61., 10., 10.}
$\endgroup$
2
  • 3
    $\begingroup$ My guess is that evaluateSingleNetworkOnOnePole or a function it calls uses n but does not localize it as it should. $\endgroup$ – Michael E2 Jan 11 '20 at 18:29
  • $\begingroup$ @MichaelE2 I agree - I also made sure to localize n in the other functions as well. $\endgroup$ – Jmeeks29ig Jan 11 '20 at 20:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.