What is Association[Rule[“ID”,number ] and how it is used?

Question

I am trying to understand the structure and the use of Dataset and I came across something new to me.

First check this example:

ds1=Dataset[{
  <|"a" -> 1, "b" -> "x", "c" -> 6|>,
  <|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
  <|"a" -> "x", "b" -> "z", "c" -> {3}|>,
  <|"a" -> 4, "b" -> "x", "c" -> {4, 5}|>,
  <|"a" -> 5, "b" -> "y", "c" -> {5, 6, 7}|>,
  <|"a" -> 6, "b" -> "z", "c" -> {}|>}]

ds2=Dataset[{
  <|"a" -> 1, "b" -> "x", "c" -> 6|>,
  <|"a" -> 2, "b" -> "y", "c" -> {2, 3}|>,
  <|"a" -> "x", "b" -> "z", "c" -> {3}|>,
  <|"a" -> 4, "b" -> "x", "c" -> {4, 5}|>,
  <|"a" -> 5, "b" -> "y", "c" -> {5, 6, 7}|>,
  <|"a" -> 6, "b" -> "z", "c" -> {}|>}]
ds1===ds2
(*False*)

When I look at the FullForm, I found that each one has different Association[Rule["ID", number] at the end of the FullForm. This number changes each time ds1 or ds2 are evaluated.

I have two questions:

1-If I have a big Dataset how can I know if they are equal or not?

2- What is Association[Rule["ID", number] and how it is used?

Answer 1

I'm interested in your question because I've never thought to look at the full form of a Dataset before! As such, I can't really say what the internal ID might be for in the Dataset object, except to say that SameQ likely fails because Dataset is likely treated internally like an Object. Each instantiation is distinct.

However, you can test for sameness of the data within a Dataset like so:

SameQ @@ (Normal /@ {ds1, ds2})

I hope at least that helps!