3
$\begingroup$

I have an unformatted binary file generated by fortran 77 and i want to import it in Mathematica. The file is formed by 5 data arrays with double precission float numbers. The problem is that seems that f77 writes some "Head records" before and after the actual numbers, difficulting the reading.

There is a related StackExchange question:

Import Fortran unformatted binary

but cannot put it to work. My aproximation to the problem so far was:

f = OpenRead["JCO1DADAP_0000", BinaryFormat -> True]
BinaryReadList[f, {"Integer64", "Real64", "Integer64", "Integer64",
"Real64","Integer64", "Integer64", "Real64", "Integer64", "Integer64",
"Real64", "Integer64", "Integer64", "Real64", "Integer64"},
ByteOrdering -> -$ByteOrdering]

Some numbers makes sense because I know the data are between [-10,10] (numerical order values - not real values: i.e. something like 1.24*10^-1244 is wrong) but there are others that are definitely wrong (like 1.24*10^-1244).

The problem could be that i'm not taking care of the "records" before and after the entries correctly as i do not know how was this file written far than i have written here. I know that fortran put "records"every time the WRITE function is called. So i do not know really if the records are each 5 elements or each one element.

For those of you that wants a sample here it is:

https://www.dropbox.com/s/bmruv7juorzpwhm/JCO1DADAP_0000?dl=0

If you have some questions or want more info, do not hesitate.

Thank you all

Edit: The arrays have dimension = 3840 *480

Edit 2: First lines of the Hexdump of the file:

0000000 00 00 00 14 00 00 0c 80 00 00 0f 00 00 00 01 e0

0000010 00 00 00 01 00 00 00 08 00 00 00 14 00 00 1e 20

0000020 3f f7 1c 71 c7 1c 23 97 3f 69 99 99 99 99 99 9a

0000030 3f a0 00 00 00 00 00 00 3f b0 00 00 00 00 00 00

0000040 3f b8 00 00 00 00 00 00 3f c0 00 00 00 00 00 00

0000050 3f c4 00 00 00 00 00 00 3f c8 00 00 00 00 00 00

0000060 3f cc 00 00 00 00 00 00 3f d0 00 00 00 00 00 00

0000070 3f d2 00 00 00 00 00 00 3f d4 00 00 00 00 00 00

0000080 3f d6 00 00 00 00 00 00 3f d8 00 00 00 00 00 00

0000090 3f da 00 00 00 00 00 00 3f dc 00 00 00 00 00 00

00000a0 3f de 00 00 00 00 00 00 3f e0 00 00 00 00 00 00

00000b0 3f e1 00 00 00 00 00 00 3f e2 00 00 00 00 00 00

00000c0 3f e3 00 00 00 00 00 00 3f e4 00 00 00 00 00 00

Edit 3: Here is a link with the file readed by an IDL program to check if the reading is good enough:

https://www.dropbox.com/s/k1z1tdeghxc0mne/pablo?dl=0

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Do you have any more information about the format? What are the sizes of the matrices? What about the records? Can you show the FORTRAN code that generated this file? $\endgroup$
    – MarcoB
    May 15, 2015 at 1:45
  • $\begingroup$ I know the dimensions of each array, the type of each array (double float) and that there is no overall header. Sadly, that's all. $\endgroup$
    – Dargor
    May 15, 2015 at 8:31
  • $\begingroup$ And could you share that information about the dimensions of each array with us? $\endgroup$
    – MarcoB
    May 15, 2015 at 8:36
  • $\begingroup$ So sorry, i thought it was in the question. I will update it and post that here as a comment. So sorry :( $\endgroup$
    – Dargor
    May 15, 2015 at 8:38
  • 1
    $\begingroup$ The arrays have dimension =3840 *480 $\endgroup$
    – Dargor
    May 15, 2015 at 9:02

1 Answer 1

Reset to default
4
$\begingroup$

Played around a bit and got to a point where I think I can let you finish it off (and make it more "automatic" and syntactically pleasing).

file = "/path/to/file/JCO1DADAP_0000"

For an unformatted Fortran 77 file I usually start by opening the stream an looking at the first "Integer32". This will determine how many bytes the record uses.

str = OpenRead[file, BinaryFormat -> True]
(* InputStream["/path/to/file/JCO1DADAP_0000", 3] *)

BinaryRead[str, "Integer32", ByteOrdering -> 1]
(* 20 *)

Note that the ByteOrdering is big endian. (Looking at your hexdump which started with 00 00 00 14 gave me this clue.) The first record will span 20 bytes. I will also expect to see 00 00 00 14 after 20 bytes again (which I do). The record will be surrounded by its size.

Now it's some guesswork. I tries a few types and "Integer32" is a good fit (I will need 5 of them to span 20 bytes). It returns:

BinaryReadList[str, "Integer32", 5, ByteOrdering -> 1]
(* {3200, 3840, 480, 1, 8} *)

Ah ha! Look at the 3840 and 480, that makes me think I am close to the correct track. As mentioned previously, the next "Integer32" should be 20, and it is:

BinaryRead[str, "Integer32", ByteOrdering -> 1]
(* 20 *)

The next "Integer32" should be the size of the next record:

BinaryRead[str, "Integer32", ByteOrdering -> 1]
(* 7712 *)

Let's read these in, my first guess was to believe you that these might be "Real64". There should be 964 of these then.

BinaryReadList[str,"Real64",964,ByteOrdering->1]//Short
(* {1.44444,0.003125,0.03125,0.0625,0.09375,0.125,0.15625,0.1875,
    <<948>>,
    14.8125,14.8438,14.875,14.9063,14.9375,14.9688,15.,395.343} *)

(the output above is the Short version). All looking good, could be some grid points, I don't know. Next "Integer32" should be 7712 and it is:

BinaryRead[str, "Integer32", ByteOrdering -> 1]
(* 7712 *)

Now for the next record size

BinaryRead[str, "Integer32", ByteOrdering -> 1]
(* 88473600 *)

which in nicely enough 3840*480*8*6, so maybe 6 blocks of 3840x480 double precision reals. I will let you try with reading it all in, but the first "row" looks ok...I am not sure what is exactly expected.

BinaryReadList[str,"Real64",480,ByteOrdering->1]//Short
(* {2.03828*10^-14,0.99,0.,0.01,1.85645,
    <<471>>,
    0.,0.892469,0.0122996,0.} *)

I suppose they could be complex...

Improve this answer
$\endgroup$
7
  • $\begingroup$ Wow, really impressed by the answer!! So much good work. I will perform a few test to check that the data makes sense. All look promising excepto the 2.032828*10^-14. That is inusual. The other strange thing is what precedes the 6 array blocks.....maybe grid points. I think that a good test is perform a density plot of one of the arrays. What do you think? $\endgroup$
    – Dargor
    May 16, 2015 at 3:12
  • $\begingroup$ But really, i'm impressed with the quality of the answer. How do you determine the big endian using the hex dump? $\endgroup$
    – Dargor
    May 16, 2015 at 3:13
  • $\begingroup$ I have found some strange pattern. I have updated the question with the result of reading the file with an IDL program that does that. In the last read of the big array of ¿6x3840x480 double precision reals? i have found that the numbers that appear in the file readed with IDL come ALWAYS after a .0. Do you know what that means? $\endgroup$
    – Dargor
    May 21, 2015 at 15:29
  • $\begingroup$ @PabloGalindoSalgado if I were to guess, I would suspect they are in fact complex doubles $\endgroup$
    – chuy
    May 21, 2015 at 16:08
  • $\begingroup$ Is strange, because people in my group read this file with an IDL program that reads the file as double precision floats. They import the file using: openr,1,'JCO1DAhyb',/f77_unformatted readu,1,rho,velx,vely,ene,tracer where rho,velx,vely,ene,tracer are double precision floats arrays of dim 3840*380 $\endgroup$
    – Dargor
    May 21, 2015 at 18:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.