16
$\begingroup$

Language` context is around for a long time. It is also heavily used (APIFunctions / FormFunctions dependencies deployment rely on it).

It is undocumented and while I understand WRI won't support it just because we know how to use it, it will be very handy to have clear usage descriptions of:

? Language`*

enter image description here

Feel free to add/edit exsiting answers, they are based on our experience rather than internal knowledge so they may not be accurate.

Index:

  • `*Definition*

    • `DefinitionList
    • `ExtendedDefinition
    • `ExtendedFullDefinition
  • `*Mutation*

    • `SetMutationHandler
    • `HasMutationHandlerQ
    • `HandleMutation
    • `MutationFallthrough
  • `*ExpressionStore*

    • `NewExpressionStore
    • `ExpressionStore

TODO:

{ `ArrayObject, `DestructureLValue, `FromArrayObject
, `FromVariableLengthInteger,  `FullGet, `GetLValueSymbol
, `InactivateExclusionsDefault, `LinkCombine, `NewArrayObject
, `PropertyItemMap, `TimeVariableLengthInteger
, `ToVariableLengthInteger, `UpdateInactivateExclusions
}
$\endgroup$
  • $\begingroup$ To be clear, is the question "what do the functions in Language`* do"? $\endgroup$ – QuantumDot Feb 14 '18 at 16:22
  • 1
    $\begingroup$ @quantumdot yes $\endgroup$ – Kuba Feb 14 '18 at 16:26
  • $\begingroup$ Will a more ambitious project be possible to document all the undocumented functions? $\endgroup$ – Αλέξανδρος Ζεγγ Sep 26 '18 at 5:07
  • 1
    $\begingroup$ @ΑλέξανδροςΖεγγ It would be hard to find motivation knowing that the answer is somewhere at WRIs desks anyway. So we end up cracking stuff we need or find interesting. You can always start such project, maybe on github because it may be too broad here. $\endgroup$ – Kuba Sep 26 '18 at 6:33
  • 1
    $\begingroup$ Language`*VariableLengthInteger functions allow one to convert an arbitrary integer to a short byte sequence and vice versa. This encoding method is used in WXF file format and described in the section Length Encoding (Varint) of reference.wolfram.com/language/tutorial/… $\endgroup$ – Shadowray May 7 '19 at 21:05
12
$\begingroup$

Language`*ExpressionStore

This is a hash-map system (without vectorized calls) that stores items without incrementing the ref count allowing them to be garbage collected naturally

Highly useful for various types of caching. First mentioned by Jason B here.

NewExpressionStore

Language`NewExpressionStore["name"]

Creates a new ExpressionStore

ExpressionStore

A cache that can be accessed via a slightly funky API:

put

Attaches a value to an expression and a key:

$store["put"[expr, key, value]]

get

Accesses a value attached to an expression by key

$store["get"[expr, key]]

remove

Removes the values attached to an expression:

$store["remove"[expr]]

listTable

Lists all cached values and their expressions:

$store["listTable"[]]

Notes

Hashing by explicit identity

This means that two copies of an expression, since they are different objects at the C++ level at which Mathematica is implemented, get hashed and thus stored differently:

Module[
 {
  store = Language`NewExpressionStore["random_store"],
  obj1, 
  obj2,
  sym
  },
 obj1 = sym[];
 obj2 = sym[];
 store["put"[obj1, "key", "value"]];
 {store["get"[obj1, "key"]], store["get"[obj2, "key"]]}
 ]

{"value", Null}

Ref counts preserved

The reference counter for an expression is not incremented by addition to a store. Thus if it is garbage collected it will be removed from the store

Module[
 {
  store = Language`NewExpressionStore["random_store"],
  obj,
  sym,
  res
  },
 obj = sym[];
 store["put"[obj, "key", "value"]];
 res = store["get"[obj, "key"]];
 obj =.; (* activate garbage collection *)
 {res, 
  store["get"[obj, "key"]]}
 ]

{"value", Null}

Edit

As provided by Jason B. here we also have "getKeys" and "containsQ". For those searching from the web, the lack of ref-count incrementing here makes this akin to a weak hash map.

$\endgroup$
10
$\begingroup$

Language`*Mutation*

First seen here, implemented by Taliesin Beynon

What do they do?

The only functions I really know about in this bunch are the ?Language`*Mutation* functions.

This is an interface that allows you to build things that work like CloudExpression. For example, I can make a CloudExpression and use AppendTo on it:

ce = CreateCloudExpression[<||>]

(* ==> CloudExpression[...]*)

AppendTo[ce, 1 -> 2]

(* ==> CloudExpression[...] *)

This is really implemented at the UpValues level, but for many things the expression itself might be too deep to use via UpValues, e.g.:

ce[[1]] = 4

(* ==> 4*)

For this they use Language`SetMutationHandler to handle what would otherwise have been a failed assignment:

obj["uuid"][[1]] = 2

(* > Set::setps: obj[uuid] in the part assignment is not a symbol. *)

(* ==> 2*)

And there are a bunch of symbols these days that use this:

ToExpression[
  Names["*`*"]
, StandardForm
, Function[Null, If[Language`HasMutationHandlerQ[#], #, Nothing], HoldFirst]
]

(*{Audio`AudioGraph, AudioStream, CloudExpression, EntityStore, 
 InitializationValue, PersistentObject, PersistentValue}*)

Language`SetMutationHandler

This is the heart of the interface. We call it like:

Language`SetMutationHandler[type, handler]

which registers a mutation handler for type. CloudExpression for instance has the handler CloudExpression`PackageScope`CloudExpressionMutate.

Here's a concrete example of this in action:

oop`mut~SetAttributes~HoldAllComplete
oop`mut[Set[f_[oop`ob[s_], p___], v_]] := Set[f[s], p]

oop`ob[sym][[1]] = 2

(*Set::setps: oop`ob[sym] in the part assignment is not a symbol.*)

(* ==> 2 *)

Language`SetMutationHandler[oop`ob, oop`mut]

oop`ob[sym][[1]] = 2

(* Set::noval: Symbol sym in part assignment does not have an immediate value. *)

(* ==> 1 *)

Language`HasMutationHandlerQ

Pretty clear from the previous section

Language`HandleMutation

This is how the MutationHandler system does its dirty work. For example:

Language`HandleMutation[oop`ob[sym][[1]] = 2]

(* Set::noval: Symbol sym in part assignment does not have an immediate value. *)

(* ==> HoldComplete[1] *)

One thing to note, if the Handler doesn't do anything, the system will keep trying to mutate it until it bottoms out:

Language`SetMutationHandler[oop`ob, oop`mut2]
Language`HandleMutation[oop`ob[sym][[1]] = 2]

(* oop`ob::mutreclim: Too many nested mutations occurred. *)

(* oop`ob::modfl: The modification oop`ob[sym][[1]]=2 could not be performed. *)

(* ==> $Failed *)

Language`MutationFallthrough

Language`MutationFallthrough seems to tell the system to abort the mutation process and raise the most recent failed result, for instance this total failure:

oop`mut3~SetAttributes~HoldAllComplete
Language`SetMutationHandler[oop`ob, oop`mut3]
Language`HandleMutation[oop`ob[sym][[1]] = 2]

(* oop`ob::modfl: The modification oop`ob[sym][[1]]=2 could not be performed. *)

(* ==> $Failed *)

is transmogrified into this basic failure we'd expect without the mutation system when using Language`MutationFallthrough:

oop`mut3[___] := Language`MutationFallthrough
Language`HandleMutation[oop`ob[sym][[1]] = 2]

(* Set::setps: oop`ob[sym] in the part assignment is not a symbol. *)

(* ==> HoldComplete[2] *)
|improve this answer|||||
$\endgroup$
  • $\begingroup$ So can we say that the mutation handling is a general way to handle modifying complex data types without actually introducing entirely new data types (i.e. overloading all functions like AppendTo to handle the new head)? $\endgroup$ – István Zachar Feb 15 '18 at 10:07
  • 1
    $\begingroup$ Sort-of. Your mutation handler function will still need to define a way to handle things like AppendTo. Rather I'd say it moves the work from UpValues to DownValues in mutating a data-type, allowing you to escape, e.g., the depth one restriction without sacrificing performance. $\endgroup$ – b3m2a1 Feb 15 '18 at 10:12
  • $\begingroup$ 1) What is the point of Set[f[s], p] as opposed to Set[f[s,p], v]? in SetMutationHandler example. 2) That may be subjective but I find it terrible to read with all those contexts and short names. What do you think about: ObjectMutation[ part_[Object[s_], spec___] = value_ ] := Set[part[s, spec], value], unless I missed the point or you disagree I could edit the post since I'm trying to parse it anyway. $\endgroup$ – Kuba Nov 12 '18 at 12:27
10
$\begingroup$

Language`*Definition*

[...] Language`ExtendedDefinition and Language`ExtendedFullDefinition are
analogous to Definition and FullDefinition but capture the definition of a
symbol in such a way as it can be reproduced in another kernel. [...]

From groups.google mathematica thread

Language`ExtendedDefinition

Usage

  • ExtendedDefinition[ sym ] generates _Language`DefinitionList all values associated with sym

  • The result can be used to load/overwrite kernel state with those values: Language`ExtendedFullDefinition[] = Language`DefinitionList[...]

Options

See ExtendedFullDefinition...

Basic example

bar[x_] := x + 2;
foo[x_] := {x + 1, bar[x]};

temp = Language`ExtendedDefinition[foo]

(* Language`DefinitionList[
     HoldForm[foo] -> {
       OwnValues -> {}, SubValues -> {}, UpValues -> {}
     , DownValues -> {HoldPattern[foo[x_]] :> {x + 1, bar[x]}}
     , NValues -> {}, FormatValues -> {}, DefaultValues -> {}
     , Messages -> {}, Attributes -> {}
     }
   ]
 *)

 ClearAll[foo, bar];

 foo[1]
 (* foo[1] *)

 Language`ExtendedFullDefinition[] = temp;
 foo[1]
 (* {2, bar[1]} *)

Possible issues

  • DefinitionList values for sym will not merge but overwrite existing state so any previous information about sym will be lost.

    ClearAll[foo, bar];
    
    foo[x_, y_] := x + y;
    
    foo[1]     (* ==> foo[1] *)
    foo[1, 2]  (* ==> 3 *)
    
    Language`ExtendedFullDefinition[] = temp;
    
    foo[1]     (* ==> {2, bar[1]} *)
    foo[1, 2]  (* ==> foo[1,2] *)
    

`ExtendedFullDefinition

Usage

Same as ExtendedDefinition but recursively gathers all relevant definitions

Options

ExtendedDefinition has them too but they are more relevant here.

  • "ExcludedContexts" which context to ignore when gathering definitions

    Very useful, by default it includes all(?) built in paclets and System` which automatically fixes a problem of accidentally saving information about $MachineId and friends: Not FullDefinition for Save. MachineID not Protected?

    Language`ExtendedFullDefinition[$MachineID] 
      (*Language`DefinitionList[]*)
    
    Language`ExtendedFullDefinition[$MachineID, "ExcludedContexts" -> {}]
      (*Language`DefinitionList[ stuff...]*)
    
  • "IssueReadProtectedMessages"

    By default False but when toggled it can help catch possible problems:

    SetAttributes[foo, ReadProtected];
    Language`ExtendedFullDefinition[foo, "IssueReadProtectedMessages" -> True]
    
    (* Language`ExtendedFullDefinition::rpsym :  Symbol foo is ReadProtected.*)
    (* Language`DefinitionList[] *)
    

Basic example

bar[x_] := x + 2;
foo[x_] := {x + 1, bar[x]}

temp = Language`ExtendedFullDefinition[foo]

(* Language`DefinitionList[
     HoldForm[foo] -> {
       OwnValues -> {}, SubValues -> {}, UpValues -> {}
     , DownValues -> {HoldPattern[foo[x_]] :> {x + 1, bar[x]}}
     , NValues -> {}, FormatValues -> {}, DefaultValues -> {}
     , Messages -> {}, Attributes -> {}
     }
   , HoldForm[bar] -> {
       OwnValues -> {}, SubValues -> {}, UpValues -> {}
     , DownValues -> {HoldPattern[bar[x_]] :> x + 2}
     , NValues -> {}, FormatValues -> {}, DefaultValues -> {}
     , Messages -> {}, Attributes -> {}
     }
   ]
*)

`DefinitionList

As far as I know it is just a symbolic wrapper returned and used by Extended*Definition functions.


Further reading:

|improve this answer|||||
$\endgroup$
  • $\begingroup$ I see a way to solve this old problem of automatic deployment of code and dependent functions for clean and simple distribution. $\endgroup$ – István Zachar Feb 15 '18 at 10:37
  • $\begingroup$ @IstvánZachar yes but in majority of cases i prefer to inject raw source packages as their setup may be environment dependent. $\endgroup$ – Kuba Feb 15 '18 at 10:45
  • $\begingroup$ @IstvánZachar and this is still undocumented :) $\endgroup$ – Kuba Feb 15 '18 at 10:56
  • $\begingroup$ You are of course right Kuba, but my main problem with packages is that my functions are distributed in various packages and deploying all related packages would mean a lot of unnecessary data and function-exposition, increasing clatter within the namespace and within the head of users. Well, to be honest, I haven't shared much code recently, so it is no big issue for me. $\endgroup$ – István Zachar Feb 15 '18 at 11:31
  • $\begingroup$ You are right too, it depends what fits your workflow best. I just happen to need to produce standalone .cdfs where users don't use functions manually anyway. $\endgroup$ – Kuba Feb 15 '18 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.