1
$\begingroup$

I'm analyzing data for Covid-19 and found an interesting website: "https://covidtracking.com/api/states/daily.csv". My problem: using SemanticImport, it seems stuck on the data that I read yesterday even though the site is returning data for today.

Running on MMA v12.1, OSX 10.14.6.

The code is simple:

Import["https://covidtracking.com/api/states/daily.csv"]
(* versus *)
SemanticImport["https://covidtracking.com/api/states/daily.csv"]

Import returns:

(*
{{"date", "state", "positive", "negative", "pending", "hospitalized", 
  "death", "total", "dateChecked", "totalTestResults", 
  "deathIncrease", "hospitalizedIncrease", "negativeIncrease", 
  "positiveIncrease", "totalTestResultsIncrease"}, {20200326, "AK", 
  59, 1801, "", 3, 1, 1860, "2020-03-26T20:00:00Z", 1860, 0, 2, 152, 
  17, 169}, {20200326, "AL", 506, 3593, "", "", 1, 4099, 
  "2020-03-26T20:00:00Z", 4099, 1, 0, 1064, 223, 1287}, {20200326, 
  "AR", 335, 1504, 0, 41, 3, 1839, "2020-03-26T20:00:00Z", 1839, 1, 
  19, 67, 55, 122},...}
*) 

indicating that there is data available for today (2020-03-26) whereas SemanticImport returns this: Dataset view from SemanticImport

Why isn't SemanticImport starting the data from 20200326 as shown by Import? I have quit out of my current session (closed Mathematica) and restarted thinking that some sort of caching is occurring but that had no effect on the result.

If I cannot figure out why these are different, I'll just have to use Import rather than using SemanticImport.

I'm using the returned data to evaluate the progress of testing for Covid-19 in graphs such as this: Testing by date (US only)

$\endgroup$
4
  • 1
    $\begingroup$ This is essentially a duplicate. I posted a similar question 8 days ago. I submitted a report to WRI. It's a confirmed bug. $\endgroup$ – david Mar 27 '20 at 1:31
  • $\begingroup$ I didn't realize it was a dup - thanks for letting me know. Perhaps the code I posted in the answer will help you until they fix the bug. $\endgroup$ – Mark R Mar 27 '20 at 1:35
  • 2
    $\begingroup$ This is a duplicate of SemanticImport vs Import differ on same CSV file? $\endgroup$ – Szabolcs Apr 23 '20 at 18:56
  • 2
    $\begingroup$ It's a bug that several of us are affected by. Please contact Wolfram about it. That should help prioritize it. $\endgroup$ – Szabolcs Apr 23 '20 at 18:56
1
$\begingroup$

The following code does what SemanticImport isn't doing correctly:

ImportCSVToDataset[url_] := Block[
  {
   allData = Import[url, "CSV"],
   headerLine,
   dataLines,
   dataLinesAssociation
   },
  {headerLine, dataLines} = TakeDrop[allData, 1];
  headerLine = First@headerLine;
  dataLinesAssociation = 
   Association[Thread[headerLine -> #]] & /@ dataLines;
  Dataset[dataLinesAssociation]
  ]

As you can see by this screenshot, it is including the data from 2020-03-26. Dataset using ImportCSVToDataset

$\endgroup$
1

Not the answer you're looking for? Browse other questions tagged or ask your own question.