Datasets and Time Values: Filtering and Plotting

Question

After struggling for a few weeks, I have accumulated a few examples of challenging date-related operatations with datasets. Some I have partial solutions; others I cannot yet do. The resources on Datetime objects in datasets is thin on here (both are relatively new to Mathematica, so no surprise), so maybe this will fill in that gap.

For all of my data, we have events that occur at a datetime. We want to understand how the progress on given days compare; for example, when did task A occur on April 4, 2017 vs October 12, 2018? What is the distribution for Task A occurring throughout a day over 100 days?

1. How can I plot a DateHistogram with a bin of, say, 20 minutes? I can do it by hour: DateHistogram[dataset, "Hour", DateReduction->"Day"]

2. How can I utilize the DateHistogram plot style using the operator form? A relatively new functionality allows the syntax dataset[plotstyle,"Key"] (see this nice StackExchange thread). For example, I can implement:

   dataset[Groupby[Key["State"]] /* (PieChart[#, ChartLabels->Keys[#]]&), Length]

   I want to use this syntax with DateHistogram; I can implement the most basic version:

   dataset[DateHistogram,"Sent"]

   I want to add the DateReduction option, along the lines of:

   dataset[DateHistogram["Hour",DateReduction->"Day"],"Sent"]

   Unfortunately, the above example doesn't work and I can't find more documentation.

3. How can I select objects in a certain time window? I figured out a method, but perhaps there is a more elegant solution. In the below example, I can plot the events that occur before 2016:

   eventsBefore2016 = dataset[Select[#Sent<DateObject[{2016,1,1}]&,"Sent"]; DateHistogram[eventsBefore2016,"Hour",DateReduction->"Day"]

   Can I make these two lines into one line of code, for example? Could I do it using the operate form (question 2)?

4. How can I adapt a DistributionChart for a dataset of Datetime values? I don't example starter code for this one.

(Apologies for the generic code examples below; I can't share the actual data and my workplace blocks the tutorial datasets such as titanic. If this is a real sticking issue I can adapt this question with a tutorial dataset later at home.)

Answer 1

Q1:

You can use Quantity[20, "Minutes"] as the bin specification:

maildata = Databin["4UYrgYkd"];
DateHistogram[Values[maildata], Quantity[20, "Minutes"], DateReduction->"Day"]

Q2:

dataset[DateHistogram[#, "Hour", DateReduction->"Day"]&, "Sent"]

Q3:

Three possible ways:

dataset[DateHistogram[#, "Hour", DateReduction -> "Day"]& @*
 Select[# < DateObject[{2016, 1, 1}]&], "Sent"] 

dataset[Select[#Sent < DateObject[{2016, 1, 1}]&] /* 
  (DateHistogram[#,"Hour", DateReduction->"Day"]&), "Sent"]

dataset[DateHistogram[Select[# < DateObject[{2016, 1, 1}]&],
  "Hour", DateReduction -> "Day"]& , "Sent"] 

Q4:

Processing maildata to get time-stamps grouped by day:

groupedByDay = KeySortBy[System`DateObjectDump`$dowAssociation]@
   GroupBy[Values @ maildata, {DateValue[#, "DayName"] &}, 
    AbsoluteTime@DateList[{2019, 1, 1, ## & @@ #[[4 ;;]]}] & /@ # &];

Note: if your dates are DateObjects, change #[[4 ;;]] to #[[1, 4 ;;]].

We can use groupedByDay with DateHistogram, DistributionChart or BoxWhiskerChart:

DateHistogram[groupedByDay, 
    ChartLayout->"Stacked", ChartStyle->3,
    ChartLegends->Keys[groupedByDay]]

For DistributionChart and BoxWhiskerChart we need to compute the date ticks for the vertical axis:

ticks = System`DateListPlotDump`DateTicks[{{2019, 1, 1, 0, 0, 
     0.}, {2019, 1, 1, 23, 59, 0.}}, 10, {"Hour", ":", "Minute"}];

BoxWhiskerChart[groupedByDay, 
  ChartLabels -> Keys[groupedByDay], 
  FrameTicks->{{ticks, Automatic}, {Automatic, Automatic}}]

If you have to use DistributionChart:

DistributionChart[groupedByDay, 
 ChartLabels -> Keys[groupedByDay], 
 FrameTicks -> {{ticks, Automatic}, {Automatic, Automatic}}, 
 ChartElementFunction -> ChartElementDataFunction["SmoothDensity", 
     "ColorScheme" -> "DeepSeaColors"]]

Use

ChartElementFunction -> ChartElementDataFunction["GlassQuantile",
   "Quantile" -> Range[0,90,10],
   "QuantileStyle" -> (Directive[Thick, Hue[#/100]] & /@ Range[0,90,10]),
   "QuantileShading" -> True]

to get

Credit: The function System`DateObjectDump`$dowAssociation is from this answer by halirutan.

System`DateObjectDump`$dowAssociation

<|Monday -> 1, Tuesday -> 2, Wednesday -> 3, Thursday -> 4, Friday -> 5, Saturday -> 6, Sunday -> 7|>

Update: Working with Dataset input:

We first construct a data set using the timestamps in maildata.

maildata = Databin["4UYrgYkd"];
timestamps = Values[maildata] /. a : {_, __} /; Length[a] == 6 :> DateObject[a];

SeedRandom[1]
from = RandomChoice[{"A", "B", "C", "D"}, Length @ timestamps];
to = RandomChoice[{"A", "B", "C", "D"}, Length @ timestamps];
subject = StringRiffle /@ Partition[RandomWord["Noun", 2 Length @ timestamps], 2];
ds = Dataset[AssociationThread[{"from", "to", "subject", "sent"} -> #] & /@
   Thread[{from, to, subject, timestamps}]]

"abuse randomness"? There is something suspiciously non-random about the subject lines:)

Create a new data set showing DateHistograms in hourly bins of daily message traffic grouped by senders and recipients:

ds2 = Transpose @ ds[GroupBy["from"], GroupBy["to"], 
  Select[#sent < DateObject[{2010, 12, 15, 1, 0, 0}]&] /* 
   (DateHistogram[#, "Hour", DateReduction -> "Day"]&), "sent"];
Labeled[ds2, {"from", "to"}, {Left, Top}, LabelStyle -> 16, RotateLabel -> True]

Pull the date histogram of messages from "A" to "C":

ds2["A", "C"]

DistributionChart of message timestamps grouped by sender:

grouped = ds[GroupBy["from"], All, "sent"];
DistributionChart[Map[AbsoluteTime] /@ grouped, ChartLabels -> Automatic]

Group timestamps by hour and use BoxWhiskerChart to show the distributions of timestamps within in each hour:

grouped2 = ds[GroupBy[DateValue[#sent, "Hour"]&], All, "sent"];
BoxWhiskerChart[Map[Dot[{60, 1}, #] & /@ DateValue[#, {"Minute", "Second"}]&] @ 
  KeySort[grouped2] , ChartLabels -> Automatic, ImageSize -> Large]