# Tag Info

58

I will answer a couple of your questions only. Space efficiency Packed arrays are significantly more space efficient. Example: Let's create an unpacked array, check its size, then do the same after packing it: f = DeveloperFromPackedArray[RandomReal[{-1, 1}, 10000]]; ByteCount[f] ByteCount[DeveloperToPackedArray[f]] (* 320040 80168 *) Time efficiency ...

49

The difference Packed arrays give you pretty much an access to a direct C memory layout, where the arrays are stored. Unpacked arrays reference arrays of pointers to their elements. This explains most of the other differences, in particular: Space efficiency: if you look at how much space is required for packed arrays, you see that it is exactly the ...

33

I will try to list some cases I can recall. The unpacking will happen when: The result, or any intermediate step, is a ragged (irregular) array. For example Range /@ Range[4] To avoid this, you can try to use regular structures, perhaps padding your arrays with zeros appropriately The result (or any intermediate step) contains numbers of different ...

20

I'll be the first to simply mention that you can use On["Packing"] and then observe any unpacking that occurs in the course of evaluation. I'm not sure there is a more systematic way to approach this, other than to compile a list of functions that do or do not preserve packing. There are always a few surprises for me, such as PadRight on a ragged array ...

17

On my system (Windows 7 64-bit, 12GB, Mathematica v8) I only see a factor of 2 between the image file size and the memory used by the image data. This agrees with the observation that packed arrays of integers use 32 bits per element. To confirm this, a ConstantArray containing values of $2^{31}-1$ (the maximum signed 32-bit integer) is packed and has a ...

14

As I suggested in my answer to a related packed-array question, the main problem is IMO not in the data structure (packed array) per se, but in all the functions which must work with this data structure together and in concert, to make it really well-integrated into the language. Notice that there isn't a separate boolean atomic type in Mathematica, True and ...

11

Did you know about the second argument of Total, which lets you sum up element at a certain level, which in practice means along a certain dimensions? For example, if you want to keep levels 1 and 2, and sum up along level 3, you can use Total[data, {3}] (* ==> {{10, 22, 34}, {50, 62, 74}} *) Or sum up along dimension 1: Total[data, {1}] (* ==> ...

11

Let me put my comment into an answer, because I think we might have misunderstood each other. You answered in the comment However, it would be great to do whole process (preparation of the matrix, solving the eigensystem, and further analysis) in Mathematica. That exactly was my idea. You only write some lines of C-Code which are compiled into a ...

11

This is a bit long for a comment, but I think it is useful information in the context of this question. No, SparseArrays cannot be packed, because a packed array and a sparse array are completely different and unrelated data structures. But, a sparse array can be constructed from packed arrays. Let's look at an example: Vb = SparseArray[ {l_, m_, s_, q_, ...

10

This question came up in Chat the other day. Here is the solution I proposed. banded[n_Integer?EvenQ] := With[ {main = RandomReal[99, n - 1], side = SparseArray[{}, n - 2, -0.5]}, SparseArray[{i_, i_} :> main[[i]], n - 1] + Sum[side ~DiagonalMatrix~ i, {i, {-1, 1}}] ] This uses several tricks and observations. Credit for the first ...

10

One potential problem with directly testing the complete list using DeveloperPackedArray[arr_, type_] is that this test may fail if the whole array is not packed, but sub-arrays are. As a result, one can too easily fall through to the more general test using FreeQ (or MemberQ) and end up unpacking anyway. This can be avoided by using ReplaceAll to remove ...

9

The problem is that when Mathematica processes myF[kmax_] = Sum[v[[k]]/k, {k, 1, kmax}] it will immediately evaluate the right-hand side. And since it doesn't have a numeric value for kmax yet, it can't evaluate the Sum yet, which leads it to tell you that in v[[k]], k is not an integer, because it's still a symbol! If you instead use a delayed set :=, ...

8

In the Developer Utilities Package there's a function PackedArrayQ that checks if an expression is a packed array, optionally of a certain type. This can be combined with the previous test: ComplexArrayQ[arr_]:=DeveloperPackedArrayQ[arr, Complex] || (Not@DeveloperPackedArrayQ[arr] && Not@FreeQ[arr, Complex]) arr = RandomComplex[1 + I, ...

8

Given the millions of times that code will be run, for a size range of about 1000, all I could think to speed up @Mr.Wizard 's code is to memoize what stays fixed i : diags[n_] := i = With[{side = SparseArray[{}, n - 2, -0.5]}, Sum[side~DiagonalMatrix~i, {i, {-1, 1}}]]; banded2[n_Integer?EvenQ] := With[{main = RandomReal[99, n - 1]}, ...

8

This is not because of ConstantArray, but is because of unpacking due to mixing different types. In short, packed arrays are more memory efficient and can contain only data of one type (Integer, Real or Complex) in regular lists (any dimension). That way, Mathematica need not store the type of each element. a = ConstantArray[0., {10000, 100}]; ...

7

While you said your example was a simplified version, if the full form is similar, I would consider using vector processing: myF[kmax_] := Total[v[[;;kmax]]/Range[kmax]] where v[[;;kmax]]/Range[kmax] is evaluated term wise, i.e. it gives $\frac{v_i}{i}$. There are a number of optimizations internally that make this faster than Sum. Note, the use of ...

7

Notice the attributes of Table - HoldAll it means that the arguments (the function to be evaluated, AND the iterators) are not evaluated prior to executing the Table[...]. Therefore, the second loop involves computations in the iterators if you change the second Table to Table[f[t, x], Evaluate[{x, 0.5 Ti, 125 Ti, 0.5 Ti}], Evaluate[{t, ht, 12500 ht, ht}]]; ...

7

We can compare the results of the two forms of GatherBy for varying data set sizes: RandomSeed[1]; ListPlot @ Table[ RandomInteger[4,{n,3}] /. set_ :> { n , Boole @ SameQ[ GatherBy[set, Sort] , GatherBy[set, ("x";Sort@#)&] ] } , {n, 1, 300} ] The x-axis shows the set size and the y-axis shows 1 ...

7

You should note that you are actually controlling the compiling and the array packing is just coupled to that and AFAIK can't be controlled independently (anymore). You can verify this with e.g. this uncompilable table body which generates the same result: DeveloperPackedArrayQ[Table[i /. x_ /; x > 300 :> RandomReal[], {i, 1, 251}]] False I would ...

7

This is not really a guideline, but it is so basic/common I felt it deserved to be mentioned. Be careful when using Set with Part << Developer packed4 = packed3 = packed2 = packed = ConstantArray[0, 100]; packed[[{1, 2, 3, 4, 5}]] = {6, 7, 8, 9, 3}; packed2[[1 ;; 5]] = {6, 7, 8, 9, 3}; packed3[[1]] = 1; packed4[[{1, 2, 3, 4, 5}]] = {6, 7, 8, 9, 3} ...

7

Thanks Mr.Wizard. I've filed a bug for this, #288440 if you have future correspondence with people inside the company about it...

6

The part about a workaround can, I think, be solved by the same trick as I very recently suggested here - prepend an idle rule: AbsoluteTiming[ Range[1000000] /. { arr_?DeveloperPackedArrayQ :> arr, x -> 0, a -> 1, b -> 2, c -> 3, d -> 4, e -> 5, f -> 6, g -> 7, h -> 8, i -> 9, j -> 10}; ] (* ...

6

As for the reason: I can't comment on the motivation for this, obviously, but it seems to be the case that replacement using a hash table (Dispatch object) is not possible with a packed array. We can see this from the fact that the threshold for unpacking is only four rules, rather than eleven, when we specify Dispatch explicitly on the right-hand side: ...

5

In addition to the distinction between = and := shown by jVincent, you can prevent Part from receiving a bad argument by introducing an auxiliary function with a restrictive pattern. This is most commonly done with the pattern _?NumericQ and you will see many examples if you search this site for NumericQ. Here I will use _Integer as that is more ...

5

I don't know why it doesn't work on your system (it does mine), but perhaps this will help: getSegments[lst_, test_] := SplitBy[lst, test][[ If[test @ First @ lst, 1, 2] ;; ;; 2 ]] getSegments[{1, -2, 3, 4, 5, -3, -4, 9, 7, 0, 8}, Positive] {{1}, {3, 4, 5}, {9, 7}, {8}} I found an older question that relates to this. It's apparently a known bug ...

5

The larger byte count in the second case is because the array is not packed. You can check if an array is packed as: Developer`PackedArrayQ /@ {arr1, arr2} (* {True, False} *) Packed arrays contain only elements of the same kind and so space is saved by not tracking the type of each element. The reason why the first case is packed whereas the second isn't ...

5

You could do something like this: list = RandomInteger[1000,1000000]; rules = Dispatch@Thread[Range[0, 1000] -> RandomSample[Range[0, 1000]]]; newlist = list /. rules; Thread creates rules for the integers from 1-1000 to a random permutation of the same integers. /.then does the replacement. Edit: I added Dispatch as per Oleksandr R.'s comment. It ...

5

I may be misunderstanding what you need, however consider this. If you have an structure of nested lists such as your data, then summing across the deepest list is easily accomplished using Map[Total,data,{-2}]. So, as long as you want to remove dimensions from "the back" you are good to go. And if we need to remove for example the second dimension, then you ...

Only top voted, non community-wiki answers of a minimum length are eligible