# How to merge Datasets representing matrices with named rows/columns?

I am working with square (non-symmetric) matrices with named rows and columns. The data I receive is sometimes messy and rows/columns may come in various permutations. I usually need to semi-manually put every matrix into a canonical permutation before I can work with it. I'd rather never have to think about permutations and not even define a canonical one, just index the matrices using the names of their rows/columns.

I was hoping that Dataset would be helpful here, and I made Dataset objects like these:

ds1 = Dataset[<|"a" -> <|"a" -> 1, "b" -> 2|>, "b" -> <|"a" -> 3, "b" -> 0|>|>]


ds2 = Dataset[<|"a" -> <|"b" -> 1, "a" -> 5|>, "b" -> <|"b" -> 1, "a" -> 2|>|>] (* I used a different permutation on purpose *)


What is the simplest way to merge these together to obtain the following?

Dataset[<|"a" -> <|"a" -> {1, 5}, "b" -> {2, 1}|>, "b" -> <|"a" -> {3, 2}, "b" -> {0, 1}|>|>]


Instead of merging with List I will usually need to merge with other operations, e.g. Plus. Once I've built a dataset with List it's easy to do that though.

-
Is this simple enough? Merge[Normal/@{ds1, ds2}, Merge[#, Identity] &] // Dataset – Rojo Aug 5 '14 at 22:48
@Rojo if you use operator forms, it can get (in my view) a bit cleaner: Merge[Merge[Identity]] @ Map[Normal] @ {ds1, ds2} – Taliesin Beynon Aug 5 '14 at 22:54
@Szabolcs I'm deliberately waiting to see what use cases are most common before I devise a general Dataset-merging operation. Feel free to propose something if you think you've nailed the semantics of what such a thing should be. – Taliesin Beynon Aug 5 '14 at 23:00
@Rojo nice. But the tricky part here will be dealing with intermediate levels that aren't associations. – Taliesin Beynon Aug 5 '14 at 23:01
@TaliesinBeynon One (simple) generalization to Merge would be to allow level specifications. I repeatedly find myself using something like Nest[Merge, fun, n] where n is the desired depth and fun is the desired lowest-level merging function. – Mohammed AlQuraishi Dec 5 '14 at 19:29

Since the question stems from working with matrices with named rows and columns I think the answer below is relevant. My answer does not use or deal with Dataset objects, but with sparse matrices with named rows and columns.

In the last few years I have used a lot the R base library Matrix that has implementation of sparse matrix objects and efficient computations. To the sparse matrices from the Matrix R library one can assign and retrieve row and column names with the functions colnames and rownames. Sometimes I miss this in Mathematica so I started a Mathematica package that implements similar functionalities. The package does not use RLink, it has purely Mathematica language implementations.

The RSparseMatrix objects of this package can deal with the permutations of row and column names. (Note that the package is not finished and in some functions the row and column names are ignored.)

This document has demos/tests. (Arithmetic operations, matrix products, joining, profiling, etc.)

This blog post has a neat application example.

This loads the package from GitHub:

Import["https://raw.githubusercontent.com/antononcube/MathematicaForPrediction/master/Misc/RSparseMatrix.m"]


Creation

Create a RSparseMatrix object with array rules (like for SparseArray) and names of rows, columns, and dimensions:

rmat = MakeRSparseMatrix[{{1, 1} -> 1, {2, 2} -> 2, {4, 3} -> 3, {1, 4} -> 4},
"ColumnNames" -> {"a", "b", "c", "d"}, "RowNames" -> {"A", "B", "C", "D"},
"DimensionNames" -> {"U", "V"}];


Show the created matrix:

rmat // MatrixForm


The creation can be done with a SparseArray object:

rmat = ToRSparseMatrix[SparseArray[rmat],
"ColumnNames" -> {"a", "b", "c", "d"}, "RowNames" -> {"A", "B", "C", "D"},
"DimensionNames" -> {"U", "V"}];

rmat // MatrixForm


Permutations

Here we permute the matrices:

MatrixForm[rmat[[RotateLeft[RowNames[rmat], 2]]]]


MatrixForm[rmat[[All, RotateLeft[ColumnNames[rmat], 2]]]]


MatrixForm[
rmat[[RotateLeft[RowNames[rmat], 2], RotateLeft[ColumnNames[rmat], 2]]]]


Converting to SparseArray

At some point it is better to convert to an inherent data structure of Mathematica:

SparseArray[rmat[[All, RotateLeft@ColumnNames[rmat]]]] // MatrixForm


-