Hot answers tagged

36

This response defines a function called traceTypes which provides a quick-and-dirty visualization of type system operation. The function is somewhat fragile as it depends upon undocumented implementation details in version 10.2. Despite this fragility, it might be useful for study purposes as it handles many common type system use cases. The code for the ...


32

Short Answer The query operator for a given level can perform both descending and ascending actions, separated by the /* operator. We can tack the descending filtering operator 1 ;; 5 onto the front of the ascending operator Total: data // Query[(1 ;; 5) /* Total, "a"] (* 15 *) The parentheses are necessary due to the tightly binding precedence of /*. ...


23

This is a corner case that occurs when ascending operators are interleaved between descending operators. This case falls into an undocumented grey area. In the Details and Options section of the Query documentation, we read the following special rule: When one or more descending operators are composed with one or more ascending operators (e.g. desc /* ...


13

As observed in the question, the problem lies with the type information of the dataset. Specifically, the type information for the innermost association is missing (i.e. "any type"). This causes the operator Query[Transpose] to fail as the type system prohibits attempting to transpose an association with an arbitrary expression. Work-arounds As noted in ...


12

I don't know if this is useful to you but it seems a little cleaner than your own code: asc = <|"z" -> 11, "x" -> 22, "b" -> 33, "a" -> 44|>; keySpan[k_Span][asc_Association] := asc[[k /. First /@ PositionIndex@Keys@asc]] asc // keySpan["x" ;; "a"] asc // keySpan["z" ;; "a" ;; 2] asc // keySpan["b" ;;] <|"x" -> 22, "b" -> 33,...


11

These work: d[Select[#[colname] > 3 &]] In the Details section of Slot, it is said that "# is equivalent to Slot[1]" so it is equivalent to write: d[Select[Slot[1][colname] > 3 &]]


11

There appears to be a bug in the type inference mechanism here. If you evaluate <<Dataset` <<TypeSystem` and then res = drugNames[byPrefix]; res // GetType (* Assoc[Atom[String], Atom[String], AnyLength] *) which is bogus. Basically that says it thinks the result is an Association with keys and values that are both String. We can compare ...


11

We must adjust the Select operator to account for the fact that the keys "a" and "b" are nested within the key "parameter": filter = Select[(#["parameter", "a"] > 1 && #["parameter", "b"] > 20) &]; We can then obtain the desired results using that filter: Query[filter] @ filenameData (* { <|"parameter" -> <|"a" -> 3., "b"...


10

Here is one way: data[All, MapAt[foo, #, #PLATFORM /. {"Dropbox" -> "LOCATION", _ -> {}}] &] Or, if one prefers If: data[All, MapAt[foo, #, If[#PLATFORM == "Dropbox", "LOCATION", {}]] &] Edit The preceding expressions use MapAt with a fixed function (foo) and a conditional set of parts (either "LOCATION" or an empty list). As @...


10

A more direct solution with Fold using Range to reverse. With rfc[f_, g_, i_Integer?Positive] := Fold[g[f[#2] /* #1] &, g[f[i]], Range[i - 1, 1, -1]] Then rfc[f, g, #] & /@ Range[4] // Column g[f[1]] g[f[1]/*g[f[2]]] g[f[1]/*g[f[2]/*g[f[3]]]] g[f[1]/*g[f[2]/*g[f[3]/*g[f[4]]]]] Hope this helps.


10

@Lee's answer shows how to circumvent the problem by using RightComposition (/*) instead of Composition (@*), but it does not explain why this works. I will try to explain the "Why" in this answer. Let's first look at the original example and see how the order is decided. The following is based on the extensive documentation of Query: Query differentiates ...


9

@SquareOne's answer shows the natural, idiomatic way to express the query. It exploits the fact that an association can be applied to a key to extract that key's value. A similar work-around would use part notation: d[Select[#[[colname]] > 3 &]] This response will show how to achieve the desired result using only Slot notation. The motivation is ...


9

As confirmed by @Stefan R, it is a bug in 10.3. In the meanwhile, a possible workaround is : Merge[xx, Mean]


9

The behaviour is by design. When we query a Dataset, the system tries to infer whether we wish the result to be a dataset itself, or just a simple value. The decision is made using some heuristics, but it essentially boils down to this: if the result is "atomic data", then it is returned directly. Otherwise, the result is wrapped back up into a Dataset. ...


9

I'm not sure I'm missing something here, but as far as I can see, you should regard this as a feature. When you evaluate your Query, one of the later steps is to decide whether or not the final result will still be wrapped in Dataset. Take your simple example and browse through the Trace. There, one thing caught my eye: the function Dataset`ReturnRawDataQ. ...


9

This appears to be another example of the WRI-acknowledged bug described in Possible bug involving Dataset/Query and RightComposition. In short, the query is being "compiled" into an expression that applies the "age" operator at the wrong level. Analysis If we "compile" the query, we can see that the descending "age" operator is being incorrectly applied ...


8

Perhaps, there is a better one, but here is one that comes to mind: ClearAll[transform]; transform["Dropbox"] := foo; transform[_] := Identity; and then data[All, With[{tr = transform[#PLATFORM]}, MapAt[tr, #, {Key["LOCATION"]}]] &]


8

Okay, this question is already answered, but I want to add my 'preferred way' here: First, we define a new operator: ApplyIf[f_, g_, x_] := If[TrueQ[f[x]], g[x], x]; ApplyIf[f_, g_][x_] := ApplyIf[f, g, x]; Then we do the query: data[All, ApplyIf[#PLATFORM == "Dropbox"&, MapAt[foo, "LOCATION"]]] I think it's pretty clear -- we're using composition ...


8

Extended comment converted to a non-answer, nailing down what is required to trigger this bug: Here's a simple Dataset with one key having all string values, another having integer values: ds = Association @@@ ({"a" -> ToString@#, "b" -> #} & /@ Range[20]) // ReplacePart[{1, 1} -> Missing[]] // Dataset If we query the integer values, then ...


8

The 2nd example given under Query > Properties & Relations tells us Before being applied, Query expressions are "compiled" into ordinary compositions of ordinary Wolfram Language functions and their operator forms. To see the compiled form of a Query, use Normal. That seems like a good line of exploration. Let's see where it leads. Query[All, "b", ...


8

A quick fix is to add TrackedSymbols :> {searchstring} to the Dynamic. But at the end it is just another example from a neverending series "Dynamic does/not fire unexpectedly". This series taught me to not use Dynamic to calculate things but only to display them. It will be more work on your side but it will dramatically reduce number of headaches. ...


7

The following query will return the desired result: h[Select[AbsoluteTime@#["basicData", "dateMMa"] < AbsoluteTime@{2014,10,8,1,0,0} &]] (* {<|"basicData" -> <|"ID" -> "1008", "dateMMa" -> {2014, 10, 8, 0, 0, 0.}|>, ... |> *) The vital construction in this solution is #["basicData", "dateMMa"]. It references the nested key ...


7

Here is another relatively straightforward approach: data[All, <|#, "LOCATION" -> If[#PLATFORM == "Dropbox", foo[#LOCATION], #LOCATION]|> &] Or using Mr.Wizard's suggestion: data[All, <|#, "LOCATION" -> If[#PLATFORM == "Dropbox", foo, # &][#LOCATION]|> &]


7

Let's go through it blow-by-blow... Query[DeleteMissing] @ d1 This will apply DeleteMissing to the list of associations. The documentation says: DeleteMissing[list] drops elements with head Missing from a list. All of the elements of the list have head Association, so this operation is a no-op. d1[All, #c &, Head] This works as advertised, ...


7

This will work: db // Query[ Drop[ #, None, {3} ]& @* Transpose, 2, 1 ;; 3 ] {<|"English" -> "a", "Greek" -> "α"|>, <|"English" -> "b", "Greek" -> "β"|>, <|"English" -> "c", "Greek" -> "γ"|>} Equivalently we could have written: db // Query[ Transpose /* ( Drop[ #, None, {3} ]&), 2, 1 ;; 3 ] or thinking positively (instead of ...


7

This should do what you want: ClearAll[KeyPatternQuery]; KeyPatternQuery[pat_ -> f_][asc_] := With[{ g = Function[{key, val}, If[MatchQ[key, pat], key -> f[val], key -> val]] }, Association@@KeyValueMap[g, asc] ] For example with asc = <|"b2" -> 2, "a1" -> 1, c3 -> 3|> and keyPattern = k_String /; StringMatchQ[k, "a*"];...


7

How about: testdb[ All, <| "LOCATION" -> "LOCATION", "NEWDATA" -> Total @* (Pick[#DATA2, Between[{5,8}] /@ #DATA1]&) |> ] If #DATA1/#DATA2 are very long, then you might want to use something like: Pick[#DATA2, Unitize @ Clip[#DATA2, {5, 8}, {0, 0}], 1]& instead of Pick[#DATA2, Between[{5,8}] /@ #DATA2]&...


6

Workaround Even though I cannot explain the behaviour, I note that the level one ascending operators will remain right-composed by means of a level one subquery or by a subsequent query: Dataset[<|x->1, y->2|>][Values /* Query[f /* g /*h], z] Dataset[<|x->1, y->2|>][Values, z][f /* g /*h] (* h[g[f[{z[1],z[2]}]]] h[g[f[{z[1],z[2]}]]]...


6

You can use Function to proceed myDs[Select[Or @@ (Function[id, #id == id] /@ {5, 4, 1}) &], "name"] but there are shorter ways: myDs[Select[MemberQ[{5, 4, 1}, #id] &], "name"] myDs[Cases[KeyValuePattern["id" -> (5 | 4 | 1)]], "name"] myDs[GroupBy[#, Key["id"] -> Key["name"], First] & /* Lookup[Key /@ {5, 4, 1}]] but what if I have ...


6

From version 10.4 onward, we can define keySpan like this: keySpan[k1_, k2_] := Replace[<|___, s:PatternSequence[k1 -> _, ___, k2 -> _], ___|> :> <|s|>] so that: $a = <| "z" -> 1, "x" -> 2, "b" -> 3, "a" -> 4 |>; $a // keySpan["x", "a"] (* <|"x" -> 2, "b" -> 3, "a" -> 4|> *) We can make this more ...


Only top voted, non community-wiki answers of a minimum length are eligible