# Possible bug in v9.0 Rest[] function

Just to be clear, this all works perfectly in v8.0.4.0. It only fails me in v9.0. I use 64-bit Linux, 64-bit versions of Mathematica, and I have 16GB of RAM.

I have a data file formatted as follows:

Date,Open,High,Low,Close,Volume,OpenInt
19700102,1.6047,1.6140,1.6047,1.6140,881445,0
19700105,1.6140,1.6235,1.6047,1.6235,549208,0
19700106,1.6140,1.6140,1.6047,1.6140,508527,0
19700107,1.6140,1.6424,1.6140,1.6424,752620,0


There are 10839 rows of data in the file. I store the first col of data in a list called "dataDate", the second col of data in "dataOpen", etc... All works great until I try to print the data back to the screen. Version 8 displays the data as expected; but version 9 borks, the kernel resets, and all function definitions and data structures are reset.

Here's the updated, simple code:

data = Import["pg.us.txt", "Data"];

(* Read the first row of data, to get the column headings *)

(* Store each column of data in a separate list, removing the first element with Rest[] *)
dataOpen = Rest[data[[All, 2]]];
dataHigh = Rest[data[[All, 3]]];
dataLow = Rest[data[[All, 4]]];
dataClose = Rest[data[[All, 5]]];
dataVolume = Rest[data[[All, 6]]];
dataOpenInt = Rest[data[[All, 7]]];
Length[dataOpen]
Length[dataHigh]
Length[dataLow]
Length[dataClose]
Length[dataVolume]
Length[dataOpenInt]


Now, if I try to display the contents of any of the seven lists, (by simply typing the list's name and evaluating it), it works perfectly in version 8. But in version 9, any attempt to evaluate a list by evaluation of its name causes Mathematica 9 to bork: all imported data is lost as all variables are undefined, user-defined functions cleared, etc. (blue text instead of black text). It takes about 6-7 seconds for the error to occur, after you evaluate the list name and it displays the truncated output.

Here are links to my updated test notebook and data file if anyone would be kind enough to see if this is just a problem with my machine, or a bug:

https://dl.dropbox.com/u/1224415/test_case.nb

https://dl.dropbox.com/u/1224415/pg.us.txt

Thanks

UPDATE: As I stated in a response below, removing the "Rest[]" function, used to remove the first element of each list, seems to fix the problem. Perhaps I'll try to remove the first element in some other manner; or perhaps try to use "Rest[]" in a three step process (wasting RAM):

dataDate = data[[All, 1]];


UPDATE 2: It doesn't matter how much I mess with the list data: as soon as I use the "Rest[]" command on it, and display the results to screen, the program borks. I'm calling this a bug and contacting Wolfram. This is what I get for being an early adopter...

-
Works here both in 8 and 9. Can you please try to create a truly minimal example to reproduce the problem (e.g. remove the If if it makes no difference and keep the Rest if it does.)? Please also try to paste the exact same code you post here back into a notebook and evaluate it that way. Since people can't seem to reproduce the problem, it'll help if you post code that can be pasted and evaluated without modification (you could start with data = Import["https://dl.dropbox.com/u/1224415/pg.us.txt", "Data"]). –  Szabolcs Dec 14 '12 at 0:57
Also, to debug, break down your code into tiny pieces and test them separately. What does data look like? What about data[[All, 1]]? What about Rest[data[[All, 1]]]? –  Szabolcs Dec 14 '12 at 1:01
Those are good ideas; and in fact that is what I'm doing. See below. I've traced the error to the Rest[] function that is supposed to remove the first element of a list. Works in v8, causing problems in v9 (for me). –  cjpembo Dec 14 '12 at 2:03
What does data[[All,1]] look like for you? Can you post it (shortened)? What does DeveloperPackedArrayQ[data[[All,1]]] give? What if you do x = data[[All,1]] then Rest[x]? What about data[[2;;, 1]]? The question still makes it vry difficult for us to see what is going wrong there. –  Szabolcs Dec 14 '12 at 3:54
Also, please do not add the bugs tag before the community could reproduce and confirm the problem. No one so far managed to and you didn't provide enough information for us to see what is going on. –  Szabolcs Dec 14 '12 at 3:58

It works fine for me both on a (more or less) pristine 64-bit Ubuntu 12.10 machine and on OS X, so I guess it's an issue on your machine. Try launching with mathematica -cleanStart and see if that fixes it. If not, I suggest you contact Technical Support for additional assisstance.

Update: As I note in the comments below, if the kernel crashes after an output is printed, it is possible this might be due to a bug in the Suggestions Bar. You can try turning of the Suggestions Bar in Edit > Preferences (or Mathematica > Preferences on OS X) and see if this helps. Obviously a crash in the Suggestions Bar is a bug which should be fixed.

In this particular case, you should find that only dataDate produces a crash. I have traced this crash to the fact there is a long sequence of exact integers, so number-theoretic functions are called make predictions. One of them crashes the kernel--I have sent this off to the relevant developer. In the mean time, here are some possible workarounds:

1) Replace the integers with date-lists. You can still compute with these, and will probably get better predictions out of this. Plus you'll be able to format them using DateString and compute using other date-related functions.

dataDate = {Quotient[#, 10000], Mod[Quotient[#, 100], 100],  Mod[#, 100]} &  /@ dataDate


2) Use N or ToString to make your dates not be integers. This may be easier/more useful depending on how exactly you use these dates, and will avoid your particular crash.

dataDate = dataDate // N

-
If it doesn't work on your machine, first try resetting Mathematica's configuration: support.wolfram.com/kb/3274 –  Searke Dec 13 '12 at 23:08
Thanks for testing it out for me. I tried the "-cleanStart" option, but it still fails. Just to be clear; you tried this on v9.0 ? And you first tried to list the contents of "dataDate" by evaluating its name; and then you tried evaluating the other lists in the same manner? You're using up-to-date Ubuntu and I'm on Archlinux... so our systems should be similar. Only thing I can think of is I'm using a SSD drive... but I'm not writing and data... –  cjpembo Dec 13 '12 at 23:45
An update: I took the the If[] statement out, but the problem persists. I then removed the Rest[] statements used to remove the first item in each column - and it seems to be working fine. Looks like Rest[] has issues on my machine. I'll look into other methods to remove the first item in a list. –  cjpembo Dec 14 '12 at 0:26
@cjpembo don't Rest until you find the problem –  Rojo Dec 14 '12 at 2:57
Yes. I made sure to use the honest-to-goodness released version of 9. Other than the set directory, I evaluated all the notebooks in order. –  Itai Seggev Dec 14 '12 at 5:12

Disabling the "Suggestions Bar" notebook interface enhancement, located in Edit -> Preferences, fixes this issue in both of my test cases: 64-bit Linux and 64-bit Windows 7. This feature is new in v9.0. I will continue to follow up with Wolfram to see if this is a bug.

-

Just out of curiosity, and following up on Szabolcs' comment, what do you see if you remove the semi-colons (as I did) in the following lines of your code?:

\$Version
data = Import["Rest.txt", "Data"]
`