# Tag Info

0

This is only a partial answer because it turns out the AWS JSON is surprisingly complicated. Anyways this might help you get there eventually. Lots of work left unfortunately. str = Import["http://a0.awsstatic.com/pricing/1/ec2/linux-od.min.js", "String"]; pos = StringPosition[str, "callback"]; obj = StringTake[str, {pos[[1, 2]] + 2, StringLength[str] - ...

5

Maybe this: dataraw = Import["data.txt", "Table", "IgnoreEmptyLines" -> True, "HeaderLines" -> 1] {{20, 1, 40, 1}, {40, 2, 80, 2}, {60, 3}} Then for example: data = PadRight[dataraw, {Automatic, Automatic}, "NA"] {{20, 1, 40, 1}, {40, 2, 80, 2}, {60, 3, "NA", "NA"}} Finally, data[[All, 3]] {40, 80, "NA"}

3

Using R/RLink Since this scenario happens fairly often when doing data analysis here is a solution using R (through RLink): Needs["RLink"] InstallR[] rres = REvaluate[ "read.table( file = \"./data.txt\", quote = \"\", header = TRUE, fill = TRUE, stringsAsFactors = FALSE)"] (* RDataFrame[RNames["s1.km", "t1.h", "s2.km", "t2.h"], RData[{20, 40, 60}, ...

2

Many thanks to @mfvonh for this answer (posted as a comment below my question). The issue was that I was trying to assign values to strings. using ToExpression to convert the strings to expressions worked beautifully. My final code is as follows: files = FileNames["*.csv", NotebookDirectory[]]; data = Import[#] & /@ files; filenames = {}; ...

1

If you don't have V.10 Associations are out otherwise @SquareOne's solution is the better choice. Anyway quick and dirty solution I've used in the past is: filenames = {"a", "b", "c", "d"}; data = {12, 20, 40, 60}; d = MapThread[Rule, {filenames, data}]; Then accessing data for a particular name is done with "a"/.d Obviously, replace filenames and ...

4

Here is a different approach (for the solution of your problem see @mfvonh solution). What I would probably do in your case would be to create one "database" containing all your data. Let's say you have your file names: files = {"G1BCM.csv", "G1EXP.csv", "G1FIM.csv", "G1IAM.csv"}; then you can create a database as simply as: (In your case, just replace ...

1

In the interest of reducing the number of questions that have been more or less thoroughly "answered in comments", I take the liberty to transform Pickett's insightful comment into an answer: Import comes with a lot of overhead. Compare for example the following two snippets: RepeatedTiming[ ClearSystemCache[]; Import["testfile.out"]; ] ...

0

In version 10.2 (perhaps going back to 10.0, but I cannot verify that), by default STL's are imported as MeshRegion objects. Assuming that casefn is a string representing the path to an STL file, you can see all the elements available for import from the file Import[casefn, "Elements"] (*{"BinaryFormat", "BoundaryMeshRegion", "Comments", ...

5

Your text data str = "# xyscan Version 3.3.1 # Date: Wed Sep 23 11:01:55 2015 # Scanned by: dimitris # Source: /media/dimitris/E6B09AEDB09AC409/Documents and Settings/Dimitris/Documents/Prony_Math/F10.png # Comment: # Format: x y -dx +dx -dy +dy -0.20155 2.90211 0 0 0 0 0.268734 2.92958 0 0 0 0 0.802326 2.98451 0 ...

5

Assumptions No fields contain a \n or \r (true for first 7000 rows) everything is encoded in ASCII (true for first 7000 rows) Observations Some of your fields contain a , (not a problem) Some of your fields contain a \ (big problem) Your data contains the following field (in FullForm, i.e. the \ is not escaped) "PSYCHOLOGIST\CLINICAL ...

5

I've noticed before that when importing text files into Mathematica, depending on how the text file encodes numbers, you might get odd results, such as expressions like -5 - 9.18301 e due to the file encoding the number as "-9.18301e-5", which is generally fine, but doesn't read correctly with ReadList[]. Instead, try Import[filename, "Table"]. Edit: I ...

2

Import does not support the time portion of DateStringFormat. (This is a deficiency in the docs if not a bug ) If you know the specific column(s) where your date appears you can map datelist like this: MapAt[DateList[{#, {"Year", "Month", "Day", " ", "Hour", ":", "Minute", ":", "Second", ".", "Millisecond"}}] & , Import[filename.csv], {All, ...

0

I found solution at this answer: How can I import a huge CSV file quickly? And also improved loading speed of my 250Mb CSV file

11

This is a complementary answer, which shows mostly how to reduce memory use rather than speed (although later I might update it to address the speed issue as well). This answer is based on an undocumented functionality, so the usual warning applies: there is no guarantee that the method suggested below will work in future versions. Using undocumented ...

18

Analysis Reading the data is not the issue. I can read the data as strings quite fast. str = OpenRead["train-7000.csv"]; (data = ReadList[str, String]); // AbsoluteTiming//First (* 0.453251 *) Memory use is modest too: data // ByteCount (* 48550344 *) It's only slightly larger than the file on disk: FileByteCount["train-7000.csv"] (* 46483707 *) ...

12

A while ago I had to do some work on large amounts of timestamped magnetometer data. Although my work only recently reached the level of 500MB, you might be able to use some of these techniques on your files. Here's a sample of my CSV file with timestamp,x,y,z values 2015-06-03T22:21:30.827Z,10.5767,2.2233,-51.9933 ...

0

Change the data to the edge format can do the work data = Import["net.csv"] edges = Map[#[[1]] -> #[[2]]&, data] Graph[edges] Which will yield the following result

5

ReadList seems to work as you intended when used instead of Get in your code: ClearAll["file1*", "file2*"] Begin["file1"] ReadList["file1.txt", "Expression"] End[] Begin["file2"] ReadList["file2.txt", "Expression"] End[] On my system, I obtain: ?file1* Alternatively you can of course check the values themselves: a (* Out: a *) ...

5

As a workaround, you can use the batch converter included with the Andor SOLIS software to convert SIF files to FITS format. FITS is not only very similar to SIF (to the best of my knowledge), but also can be loaded into Mathematica without any problem. You will notice that the file sizes are virtually the same for both formats. When I visited Andor's ...

1

I said in a comment that is easy to change directed edges into undirected edges. Here is an example offered as proof-of-concept. g = Graph[{DirectedEdge[a, b], DirectedEdge[b, c], DirectedEdge[c, a]}] h = Graph[EdgeList[g] /. DirectedEdge -> UndirectedEdge] h//FullForm FullForm[ Graph[{a, b, c}, {UndirectedEdge[a, b], UndirectedEdge[b, c], ...

1

g = Graph[{1 -> 2, 2 -> 3, 3 -> 1}, VertexLabels -> "Name"]; h = UndirectedGraph[g, VertexLabels -> "Name"]

Top 50 recent answers are included