Questions tagged [streams]
A source of input or output: files and pipes are both examples of general Mathematica objects known as streams.
8
votes
2answers
200 views
Simple way to highlight streams in basins of attraction in StreamDensityPlot
In making a figure to answer this question, I wanted to highlight streams that start or end at the critical points, $(0,0)$ and $(2,2)$.
One can define StreamPoints or VectorPoints, but that doesn't ...
0
votes
0answers
32 views
How to force Close of opened stream when stream write previously failed (bug in opening stream twice)?
I opened a stream like:
str = OpenWrite[NotebookDirectory[] <> "myStream.tsv"];
and made an error when writing the first line (meant to be the tsv headers):...
9
votes
2answers
79 views
What is the fastest way to read the new lines in the dynamically updatable file?
I'm writing a log to a simple .txt file and want to read the last few records using Mathematica. But there are some problems.
f = OpenRead["logs.txt"];
Skip[f, ConstantArray[Number, 2], 1];
Read[f, ...
2
votes
1answer
45 views
Streams in parallel kernels. Writing in multiple files in ParallelDo loop in Mathematica
I need to write data in multiple files in ParallelDo loop:
...
2
votes
2answers
33 views
Naming indexed output streams
I need to write into multiple files in a Do loop.
Hence, I tried to open multiple streams:
...
3
votes
2answers
126 views
Join two datasets with a function similar to Vlookup [Big Dataset and Efficiency issue] - Wizards are welcome!
I am working on a huge dataset and I have a lot of data formatted in XML, we are talking about a couple of GBs. For this reason I am currently using the stream function to read them and extract the ...
2
votes
0answers
46 views
Why does ReadString complain about the stream not being opened in binary format?
This works fine:
...
10
votes
0answers
92 views
LibraryLink: How to read from a Mathematica stream in C?
Mathematica has a flexible streams interface. Functions like Read can read from a "stream" that may be backed by a file, pipe, string, network resource, etc. We can ...
3
votes
1answer
92 views
How many records? In a file (streaming)
I want to know how many records a file (large data) contains...
with streaming...like this:
...
6
votes
0answers
224 views
Communicating with server over Websocket using SocketConnect
I am trying to connect to a web socket feed of Gdax. The documentation is here, https://docs.gdax.com/#websocket-feed. There is no account needed and the information is public. I have successfully ...
0
votes
0answers
62 views
Is it possible to parallelize reading in Mathematica?
I have a gigantic data file that needs to be processed. I am reading it in chunks using ReadList with an InputStream. In my ...
3
votes
2answers
75 views
Explicitly control newline behaviour when writing strings with arbitary encoding
On Windows system, Mathematica by default opens stream which will automatically transform any "\n" into "\r\n". But I want pure ...
17
votes
1answer
337 views
Is building an event-based parser possible/reasonable in Mathematica (WL)?
Too long? Go to Problems section
Background
I like to learn by doing small, sometimes artificial, projects. Among other things I had 'parsers', and 'stream methods / streams in MMA' on my to learn ...
5
votes
1answer
167 views
Matrix Vector multiplication for very big matrices (>>RAM)
Problem
Suppose a matrix-vector product of the form $M\vec{v}$ should be calculated where the amount of storage needed to store $M$ is noticeably larger than the available RAM on a machine. What is ...
1
vote
0answers
24 views
How to export result from SQLResultSet into a csv
I have a sql query which produces very large data from a remote server but the data is too large to fit into memory all at once. So i want to read and export it sequentially into a csv. I am using the ...
1
vote
1answer
33 views
Read not advancing StreamPosition [closed]
[MMA 11.0, Win7 64bit]
I have a number of CVS files of ~1-2GB, millions of rows, so I want to read them in records, chunks building on Albert Retey's answer to this question
I assumed, and the ...
5
votes
0answers
52 views
Why ReadList ignores NullRecords for Number?
When thinking on this recent question the immediately obvious solution which came to my mind was to use ReadList with options ...
6
votes
1answer
120 views
Is there a way to redirect the output from `Echo`?
The newish function Echo is tremendously useful in quick-and-dirty debugging, since you can stick it pretty much anywhere in an expression and get feedback about ...
3
votes
0answers
32 views
Unexpected behaviour with WordSeparators and Read [duplicate]
I am confused about how Read separates elements.
The following all work fine:
...
4
votes
1answer
40 views
How to get the standard error stream for a ProcessObject?
When you run an external program using RunProcess, the program will run to completion and you will receive an association containing the exit code, stuff that went ...
5
votes
1answer
147 views
wolframscript creates tons of log files
On Linux, wolframscript seems to create log files in .Mathematica/ApplicationData/ProcessLink/Streams
for example:
...
1
vote
0answers
77 views
Why does using BinaryWrite make my file bigger? [closed]
I have a binary file 00010005.dat. I want to normalize the matrix so I use the following code:
...
3
votes
1answer
393 views
SliceStreamPlot3D or StreamPlot3D
I know that neither of the two functions listed in the title exist, but I basically replaced vector for stream to give you an idea of what I want. The most ideal would be ...
3
votes
0answers
566 views
How to visualize the vorticity of a 3D velocity field? [closed]
I sample a snapshot from a system with 50 particles including their position and instant velocities.
The snapshot is a list of 3 position and 3 velocities values per particle. i.e. 50 lines each of ...
9
votes
2answers
221 views
Is there a way to create ZIP OutputStream ?
I want to write data to a file using compression. Is it possible to do that just in time (like in java ZipOutputStream). E.g. what I want:
...
27
votes
3answers
445 views
Can Mathematica edit large binary files in-place?
I have a very large binary data file (several hundred GB) which I would like to edit using Mathematica. The only edits required are a few isolated byte replacements—no insertions or deletions. Is it ...
3
votes
0answers
135 views
How to flush stdout stream?
I am aware that one can write to a terminal standard output like so:
WriteString["stdout","Some stuff","\r"];
Note here that I am using a carriage return on ...
3
votes
1answer
107 views
How to read *all* input from stdin?
The answers given to this similar question won't do, because those solutions read only one line from stdin.
I want to read all available lines, pretty much in the ...
4
votes
3answers
525 views
how do I to create a function similar to feof in C?
Using C language,I can use feof to test if the pointer at end of file.
while(!feof(fp))
{
//to do
}
But in the mma,I ...
3
votes
1answer
165 views
How to Terminate Read at EndOfFile
We are working on a project that reads in a stream, extracts information from it, and then needs to stop the calculation when it reaches EndOfFile.
So for example,...
6
votes
1answer
65 views
Can Read[] apply a custom function automatically to the read values?
Read[stream, Number] will return a single read number. We can use a number of standard, documented types instead of Number, ...
11
votes
0answers
359 views
How to read and write data to open socket?
I am trying to open connection with socket. I am using this:
...
1
vote
1answer
96 views
Inconsistency of WriteString
Important Note
A kernel or maybe Front End restart probably fixed the problem. I did something weird today which is evaluate Context[StringQ]="test`", perhaps that ...
3
votes
1answer
86 views
`ReadString` and `StreamPosition` behaving strangely with custom terminators
I've stumbled on a strange behavior while trying to use custom terminators for ReadString.
When I use a simple String as the ...
5
votes
1answer
90 views
How can I dynamically display a stream position?
If I create a stream,
str = StringToStream["how can I dynamically display the stream position"];
Read[str, Word]
"how"
then, when I use,
...
4
votes
1answer
207 views
Importing large datasets using streams
I work with large data sets with 22 columns and up to a million rows. Each row contains one observation with several variables. All variables are numbers.
Simplyfied example:
1, 1, 2, 3, 4, 5, 6,...
4
votes
2answers
181 views
Parsing tab delimited file
Problem Description
I have my data arranged in a tab delimited file and I'm trying to parse/access certain parts of it.
As an example, I'd like to extract the number of points and the sampling ...
2
votes
1answer
95 views
Use e notation in Stream output
I'm trying to save the results of a huge Radia calculation. I evaluate the function in a loop at multiple locations.
To export the data my first approach was to fill a Table and use the ...
2
votes
0answers
919 views
StreamPlot in 3 Dimensions
I would like to plot the vector equation $\left( {\begin{array}{*{20}{c}}
{\frac{{ - a + \sqrt {{a^2} + 4bc} }}{c}}\\
{\frac{{ - a - \sqrt {{a^2} + 4bc} }}{c}}\\
1
\end{array}} \right)$ as a stream ...
2
votes
1answer
311 views
3
votes
3answers
280 views
ReadLine[file] after reopening a closed stream doesn't read at the beginning of file?
Take a text file from your system:
file = OpenRead["path/to/your/file"]
ReadLine[file]
Close[file]
Here ReadLine[file] ...
30
votes
2answers
576 views
Reading from a socket stream
I noticed an undocumented socket package (SocketLink) in Mathematica (Using version 10, not sure when it was actually added) and wrote a barebones HTTP responder ...
1
vote
1answer
263 views
ReadList can't read “.csv” data with correct line number?
Well, the problem was solved today.
Since "testAll.csv" is generated by Java/Eclipse, and I noticed that the LineSeperator is "\...
1
vote
3answers
1k views
How to import files with headers?
R has excellent header supports such as read.table with header options. I don't want to mess up with text-processing or stream-processing. I am looking for a similar tool as in R because it saves a ...
3
votes
0answers
69 views
ScheduledTask output redirection [duplicate]
When a function is run repeatedly within a ScheduledTask its output is sent to Messages Window. When two or more ScheduledTasks are running their outputs are mixed in the Messages Window and this is ...
4
votes
2answers
170 views
How to read from /dev/random?
I am looking to read a few bytes from /dev/random on OS X and Linux. The simplest approach fails on both operating systems (see below).
Why? What is a workaround?...
13
votes
3answers
717 views
ToExpression fails with long nested string expressions
Consider the following conversion from String to Mathematica Expression:
...
3
votes
1answer
125 views
EndOfFile error
I have a LARGE file (14.6 GB) that I want to separate into several smaller files. Because of its size, using Import/Export is not feasible, I think the most reasonable way of handling this would be ...
2
votes
0answers
26 views
In a Function, how to Ensure Code Runs Prior to an Abort [duplicate]
Is there a way to set aside some code in a function that will be triggered to run if the evaluation of the function is aborted by a user? Specifically, I have a function that has to open a log file to ...
4
votes
3answers
13k views
How to export data to a plain text file?
I am preparing data used by a command line program. It requires data in a simpliest possible format:
value_x1 value_y1
value_x2 value_y2
...
value_xN value_yN
...
