I'm looking for a Wolfram Q&A system that scales to 100k of text documents.

Using Mathematica's similarity search (e.g. Nearest) or vector database operations (does it have any?) along with the new OpenAI-powered function LLMSynthesize, it should be possible to build a nice Q&A system based on LLMs.

Minimum working example of idea:

index = CreateSearchIndex[{
query = "Does alice have a cat?";
qw = DeleteStopwords@TextWords[query];
deleteYellow = (StringReplace[ToString@#, 
      Shortest["Highlighted[" ~~ s__ ~~ "]"] :> s] <> "\n") &;
ctx = StringJoin[
   Flatten@TextSearch[index, Alternatives @@ qw][All, 
 StringJoin["Answer this question:", query, 
  "\n\nBased on the following context:", ctx]]

enter image description here

  • 2
    $\begingroup$ To do this offline (with short texts) you can do this with GPT2 (the section on Sentence Analogies shows how to get the sentence embeddings) To do this using OpenAI GPT3.5/4 embeddings...Christopher Wolfram's OpenAILink paclet can do this (oddly, the LLM support in 13.3 doesn't include it) In either case, take the CosineDistance in your Nearest function $\endgroup$ Aug 7, 2023 at 19:34
  • 1
    $\begingroup$ In the spirit of @JoshuaSchrier comment -- to do this with Google's PaLM you can use the paclet "AntonAntonov/PaLMLink". $\endgroup$ Aug 7, 2023 at 20:35
  • 1
    $\begingroup$ @user5601 Yes, see FeatureExtract. (There are sections discussing feature extraction from texts.) $\endgroup$ Aug 10, 2023 at 19:18
  • 1
    $\begingroup$ Nono I'm talking about retrieval not feature extraction $\endgroup$
    – user5601
    Aug 10, 2023 at 22:58
  • 1
    $\begingroup$ You're asking for simple RAG right? $\endgroup$
    – M.R.
    Aug 18, 2023 at 11:06

1 Answer 1


(This is an abbreviated version of a blog post that I wrote on the topic...)

The essential process is:

  1. Define a collection of texts and pre-process them so that they are a suitable length for the LLM context window

  2. Use the LLM to generate embedding vectors for each text item and store it for later use.

  3. When a new question is posed, generate its embedding vector and then find the Nearest vector in the collection of texts computed in #2; use this to retrieve relevant texts

  4. Send the question and the retrieved text to an LLM to generate the response


As an example, we will use the English language version of the Catechism of the Catholic Church. In addition to being good for your soul, it is also a large document that comes with a built-in segmentation into smaller pieces, is mostly plain-text HTML with minimal formatting, and is available for free online.

We'll retrieve the index page, and then use the links in the page to retrieve the remainder of the content; this example is small enough that we can store everything in memory:

sectionLinks = With[
    {url = "https://www.vatican.va/archive/ENG0015/_INDEX.HTM"}, 
    Drop[#, 10] &@Import[url, "Hyperlinks"]]; (*first 10 links are navigation tools*)

texts = Import[#, "Plaintext"] & /@ sectionLinks;

Some statistics about the documents:

Length[texts] (*how many documents do we have? *)


wordCounts = WordCount /@ texts;
MinMax[%](*how long are they?*)

(*{72, 6670}*)

GPT-style models have a limit on the number of input tokens. A rule of thumb is that a token is roughly 0.75 words (or a word is on average 1/0.75 = 1.3 tokens). For GPT-3.5, the maximum input content lengths is 8191 tokens, so we'll want to make sure that the texts that we provide are less than this, otherwise we will get an error when we generate the embeddings, which will look something like this:


Tools like tiktoken can be used to count the tokens in an input. We'll arbitrarily set 4000 words as a reasonable place to divide the text:

notTooLong[str_String] := WordCount[str] < 4000 
splitWords[str_?notTooLong] := str 
splitWords[str_] := Module[{words, half}, 
   words = StringSplit[str]; 
   half = Ceiling[Length[words]/2]; 
   {StringJoin[Take[words, half]], StringJoin[Drop[words, half]]}]

This finds the four long sections and chops them in half. Of course, in production, you might do something smarter.

Length[(shorterTexts = splitWords /@ texts // Flatten)]


Compute Embedding Vectors

Next, compute the embedding vectors for each of these texts using the LLM model of your choice:

We'll do this using the OpenAI embeddings, so we must first setup the OpenAI Link paclet the first time:


Then we import it and use it to make the embedding call:


embeddings = OpenAIEmbedding /@ shorterTexts;

Note: This will cost about $0.05 USD and it takes a minute to make all the API calls sequentially, so you may wish to save these for later reuse.

Now we'll also use these to create a NearestFunction that can be applied to future data. It is standard practice to use the CosineDistance when comparing embedding vectors.

lookupFn = Nearest[embeddings -> shorterTexts, DistanceFunction -> CosineDistance]

If you were scaling this up, you would probably want to store the texts in a database and have the Nearest function just return keys that would be used for lookup. Commercial offerings like pinecone.io can handle this for you.

Vector Retrieval

Next we take a question, generate its embedding vector and then use it to find the nearest relevant text. Let's try it:

question = "What must I do to have eternal life?";
questionEmbedding = OpenAIEmbedding[question];
closestText = lookupFn[questionEmbedding]

(*{"HelpCatechism of the Catholic ChurchIntraText - TextPART THREE: LIFE IN CHRISTSECTION TWO THE TEN COMMANDMENTSIN BRIEF Previous - NextIN BRIEF  2075 \"What good deed must I do, to have eternal life?\" - \"If you would enter into life, keep the commandments\" (Mt 19:16-17).  2076 By his life and by his preaching Jesus attested to the permanent validity of the Decalogue. 2077 The  gift of the Decalogue is bestowed from within the covenant concluded by God with his people. God's  commandments take on their true meaning in and through this covenant.  2078 In fidelity to Scripture and in conformity with Jesus' example, the tradition of the Church has always  acknowledged the primordial importance and significance of the Decalogue.  2079 The Decalogue forms an organic unity in which each \"word\" or \"commandment\" refers to all the others  taken together. To transgress one commandment is to infringe the whole Law (cf Jas 2:10-11).  2080 The Decalogue contains a privileged expression of the natural law. It is made known to us by divine  revelation and by human reason.  2081 The Ten Commandments, in their fundamental content, state grave obligations. However, obedience  to these precepts also implies obligations in matter which is, in itself, light.  2082 What God commands he makes possible by his grace.  Previous - NextCopyright (c) Libreria Editrice Vaticana"}*)

You'll notice that the text retrieved seems relevant but includes line numbers and other formatting information.

Answer the query using the retrieved text

Finally, we provide our input question and the retrieved the closestText to the LLM; Here we will use the built-in LLMSynthesize function, and a fairly minimal prompt:

   "Answer this question: ", question, 
   "\n Based on the following context:", closestText]]

(*"According to the Catechism of the Catholic Church, to have eternal life, one must keep the commandments. Jesus stated, \"If you would enter into life, keep the commandments\" (Mt 19:16-17). The Decalogue, which includes the Ten Commandments, holds a privileged expression of the natural law, and obedience to these precepts implies obligations in both grave and light matters. It is also important to note that God's grace makes it possible to fulfill His commands."*)

As you can see, this has reworded and paraphrased the retrieved text to answer the question in a satisfactory. Because we know what text was used, we could also provide a citation, if desired. Ite, missa est.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.