5
$\begingroup$

Using WL 12.1.0. Is SemanticImport caching data somewhere? I did this SemanticImport for the first time, yesterday. Worked fine. Today it misses the last column (date). However, Import correctly retrieves everything including the last column. Is there a way to force SemanticImport to get the latest data? I know I can work around by using Import, but I'd like to use SemanticImport so that I can work with Dataset. Here is an excerpt of my notebook:

SemaniticImport drops the last column in the following import:

cv19Deaths = 
  SemanticImport[
   "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/\
csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-\
Deaths.csv"];

Whereas Import correctly imports the last column:

cv19DeathsCSV = 
  Import["https://raw.githubusercontent.com/CSSEGISandData/COVID-19/\
master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-\
covid-Deaths.csv"];

Looking at the Keys:

cv19Deaths[1] // Normal // Keys

Out[]:= {"Province/State", "Country/Region", "Lat", "Long", "1/22/20", \
"1/23/20", "1/24/20", "1/25/20", "1/26/20", "1/27/20", "1/28/20", \
"1/29/20", "1/30/20", "1/31/20", "2/1/20", "2/2/20", "2/3/20", \
"2/4/20", "2/5/20", "2/6/20", "2/7/20", "2/8/20", "2/9/20", \
"2/10/20", "2/11/20", "2/12/20", "2/13/20", "2/14/20", "2/15/20", \
"2/16/20", "2/17/20", "2/18/20", "2/19/20", "2/20/20", "2/21/20", \
"2/22/20", "2/23/20", "2/24/20", "2/25/20", "2/26/20", "2/27/20", \
"2/28/20", "2/29/20", "3/1/20", "3/2/20", "3/3/20", "3/4/20", \
"3/5/20", "3/6/20", "3/7/20", "3/8/20", "3/9/20", "3/10/20", \
"3/11/20", "3/12/20", "3/13/20", "3/14/20", "3/15/20", "3/16/20"}

cv19DeathsCSV[[1]]

Out[]:= {"Province/State", "Country/Region", "Lat", "Long", "1/22/20", \
"1/23/20", "1/24/20", "1/25/20", "1/26/20", "1/27/20", "1/28/20", \
"1/29/20", "1/30/20", "1/31/20", "2/1/20", "2/2/20", "2/3/20", \
"2/4/20", "2/5/20", "2/6/20", "2/7/20", "2/8/20", "2/9/20", \
"2/10/20", "2/11/20", "2/12/20", "2/13/20", "2/14/20", "2/15/20", \
"2/16/20", "2/17/20", "2/18/20", "2/19/20", "2/20/20", "2/21/20", \
"2/22/20", "2/23/20", "2/24/20", "2/25/20", "2/26/20", "2/27/20", \
"2/28/20", "2/29/20", "3/1/20", "3/2/20", "3/3/20", "3/4/20", \
"3/5/20", "3/6/20", "3/7/20", "3/8/20", "3/9/20", "3/10/20", \
"3/11/20", "3/12/20", "3/13/20", "3/14/20", "3/15/20", "3/16/20", \
"3/17/20"}

(cv19Deaths[1] // Normal // Keys) == cv19DeathsCSV[[1]]

Out[]:= False

Update: As of 3/21/2020, using SemanticImport only pulls in data through "3/16/20" (that was the 1st time I used SemanticImport on this remote file). Unless anyone other opinions, I'm inclined to think this is a bug and will report it to WRI.

This is the workaround I'm using:

cv19DeathsCSVFile = 
  Export["time_series_19-covid-Deaths.csv", 
   Import["https://raw.githubusercontent.com/CSSEGISandData/COVID-19/\
master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-\
covid-Deaths.csv"]];
cv19Deaths = SemanticImport[cv19DeathsCSVFile];

Update 03/24/2020: Johns Hopkins CSSEGIS has deprecated the URL that I used when I created this post. They are no longer updating those files. The new URL is as specified in the following example:

cv19Deaths = 
  SemanticImport[
   "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/\
csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_\
deaths_global.csv"];

Since this was changed only yesterday, it will take another day or so to retest for the issue. I'll post when I get new results.

Update 03/25/2020: The issue persists. SemanticImport is stuck on 03/23/2020. I've submitted a bug report to WRI.

Update 03/26/2020: WRI confirmed this is a bug.

Improve this question
$\endgroup$
8
  • $\begingroup$ I can reproduce this as well, both in M12.0 and M12.1. I did not download the Deaths data yesterday, and that works fine. I did download the Cases data and there I see the same problem that you describe. As I remember, yesterday I used 12.0, not 12.1 $\endgroup$
    – Szabolcs
    Mar 18, 2020 at 15:49
  • $\begingroup$ Same here. Though, I haven't checked 12.0, yet. $\endgroup$
    – david
    Mar 18, 2020 at 18:06
  • $\begingroup$ Here is how I work around the issue: cv19DeathsCSVFile = Export["time_series_19-covid-Deaths.csv", Import["raw.githubusercontent.com/CSSEGISandData/COVID-19\ master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-\ covid-Deaths.csv"]]; cv19Deaths = SemanticImport[cv19DeathsCSVFile]; $\endgroup$
    – david
    Mar 18, 2020 at 20:38
  • 2
    $\begingroup$ I would just use Import[file, "Dataset"] if you want to work with Dataset. CSV has been updated quite a bit in Import lately but SemanticImport hasn't received these updates yet. $\endgroup$
    – GenericAccountName
    Mar 19, 2020 at 18:38
  • $\begingroup$ @GenericAccountName Can you confirm that Import is not affected by the same issue? (It's a bit inconvenient to test because it would require trying it on subsequent days.) $\endgroup$
    – Szabolcs
    Mar 19, 2020 at 19:23

1 Answer 1

Reset to default
6
$\begingroup$

If your only goal is to get a Dataset, then just use

cv19DeathsCSV = Import[
    "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_19-covid-Deaths.csv",
    "Dataset",
    "HeaderLines" -> 1
]

I can't tell you why SemanticImport is doing what it does, other than it uses heuristics to be "helpful". I prefer importing the data as is and then manipulating it myself in Mathematica.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Can you confirm whether SemanticImport will actually cache data downloaded the previous day? That would be very surprising (i.e. bad UX). $\endgroup$
    – Szabolcs
    Mar 20, 2020 at 7:20
  • $\begingroup$ This does look like it pulls in all the columns. Unfortunately, it looks like this method pulls in each row as a Dataset under a top level Dataset. It's not as convenient to use in that form. $\endgroup$
    – david
    Mar 21, 2020 at 12:50
  • 1
    $\begingroup$ @Szabolcs I took a quick look at the code (I don't work on SemanticImport). It's certainly not caching anything, but it does use temporary files for some things. Maybe there's a cleanup problem. I'd certainly report the issue to support. $\endgroup$
    – Itai Seggev
    Mar 23, 2020 at 18:18
  • $\begingroup$ @david I've updated my answer to address this. SemanticImport automatically identifies the first line as a header. Import doesn't attempt to do this, but you can tell it to about headers with an option. $\endgroup$
    – Itai Seggev
    Mar 23, 2020 at 18:31
  • $\begingroup$ @ItaiSeggev OK, I like that as an alternative. I think I'd like to wait a bit before accepting to see if someone might answer my original question. I did up-vote you for the good work around. I might use this for most of my Import's in the future as you suggest. Thanks. $\endgroup$
    – david
    Mar 23, 2020 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.