Plotting reordered clusters with different colors and joined lines

Question

I would like to show how typical days of energy load (found through clustering of a years worth of data) are spread throughout a year in a single graph with a color for each 'typical day' (or cluster).

What I have so far

Here is the data in pastebin.

I divided the daily load curves (365 * 24 measurements) into clusters:

loadclust = FindClusters[load, num]

Each list within a cluster represents a daily load curve within a year, but each cluster doesn't contain the same amount of days.

To these daily loads (lists) I then joined a list which corresponds to the hours in a year (i.e. 1 to 24 for first day, 25 to 48 for second, etc...). I did this by finding the position of each daily list in the original ordered list, since I wanted to keep the different clusters in order.

clustyearh = Flatten[Table[
Table[ 
 Join[
  Range[(Flatten[Position[load, loadclust[[j, i]], 1 ]] - 1)*24 + 
    1, Flatten[Position[load, loadclust[[j, i]], 1 ]]*24]
  , {loadclust[[j, i]]}
  ], {i, Length [loadclust[[j]]]}
 ], {j, Length[loadclust]}
], 1];

Then I made pairs of the hour in the year and the load at that hour, while still maintaining the order of the clusters (so there are fours lists containing the paired coordinates for each cluster).

pairs = Table[ 
Transpose[
 Join[{Flatten[
    Transpose[
      clustyearh[[;; Length[loadclust[[i]]]]]][[1]]]}, {Flatten[
    Transpose[clustyearh[[;; Length[loadclust[[i]]]]]][[2]]]}]]
, {i, 4}];

The result is a list of four lists of pairs {{{1, 1368},... {24, 1428}, {121, 1321}, ...} {clust2} {..}{..}}

Plotting

This is where I am having problems. First, when I used Listplot for pairs, it doesn't display all the points (I notice that it's the blue and red clusters especially missing). Second, when I include Joined -> True it joins the consecutive days in each cluster, but not in the year. If cluster one has days 1, 2, 5, 59 for example it will join those days.

Question

Is there a way to color each cluster within a line graph for the whole year? As an addition, I would love to be able to apply Manipulate to scroll through the year within a week or 2 week window.

Using Simon Wood's solution :

Create a list of indexes for the clustered days.

loadclust = FindClusters[load -> Range[365], 4];

Then make pairs of data to give as coordinates to a ListPlot, while adding data for a 25th hour for each day (i.e. the first hour of the next day) so the plot will be joined at all points:

With[{load2 = Append[load, {{0}}]},
  pairs = Map[
    Sequence @@ 
      Thread[{24 (# - 1) + Range[25], 
        Append[load2[[#]], load2[[# + 1, 1]]]}] &, loadclust, {2}]];

Create a manipulate graph to view the graph inside of a window (I have added the colors I originally used for each cluster).

Manipulate[
 ListLinePlot[pairs, Frame -> True, Axes -> False, 
   PlotRange -> {{n, n + win - 1}, All}, 
   PlotStyle -> {Blue, Orange, Red, Green}] /. 
   Line[pts_] :> Line[Split[pts, #2[[1]] - #1[[1]] == 1 &]], {{n, 1, 
   "First Hour"}, 1, 24*365 - win, 1}, {{win, 168, "Window Length"},2, 336, 1}]

Answer 1

It is probably easier to keep track of the days in each cluster if you use FindClusters like this:

loadclust = FindClusters[load -> Range[365], 4];

This returns the clusters in terms of day numbers instead of the full load curve for that day, e.g.

loadclust // Shallow

{{1, 5, 10, 15, 23, 26, 30, 32, 36, 41, <<75>>}, {2, 3, 6, 9, 11, 12, 27, 28, 29, 34, <<103>>}, {4, 8, 13, 14, 16, 20, 21, 22, 24, 25, <<85>>}, {7, 17, 18, 19, 33, 37, 58, 60, 73, 79, <<62>>}}

So the first cluster contains days 1, 5, 10 etc...

We can now replace each day number with the explicit list of {hour, load} pairs:

pairs = Map[Sequence @@ Thread[{24 (# - 1) + Range[24], load[[#]]}] &, loadclust, {2}];

The scrolling plot can be achieved by changing the PlotRange in a Manipulate. To prevent non-consecutive days from joining up you can use a replacement rule to Split each line wherever adjacent points are not 1 hour apart.

Manipulate[
 ListLinePlot[pairs, Frame -> True, Axes -> False, 
 PlotRange -> {{n, n + win - 1}, All}] /. 
  Line[pts_] :> Line[Split[pts, #2[[1]] - #1[[1]] == 1 &]],
 {{n, 1, "First Hour"}, 1, 24*365 - win, 1},
 {{win, 168, "Window Length"}, 2, 336, 1}]

Update

The simplest way to get the lines to join up is to extend each day's data to 25 hours - i.e. including the first hour of the following day.

With[{load2 = Append[load, {{0}}]}, 
  pairs = Map[
    Sequence @@ 
      Thread[{24 (# - 1) + Range[25], 
        Append[load2[[#]], load2[[# + 1, 1]]]}] &, loadclust, {2}]];

Conceptually neater is to plot the whole year as a single data set, with a ColorFunction that changes day by day to reflect the cluster containing that day.

To do this we will need a function which takes a day as its argument, and returns a colour based on cluster membership. This defines 365 DownValues for daycolour:

MapIndexed[(daycolour[#1] = ColorData[1][#2[[1]]]) &, loadclust, {2}];

The data for the ListPlot is just the complete list of {hour,load} pairs for the whole year:

data = Thread[{Range[24*365], Flatten[load]}];

Here is the scrolling plot again, using the cluster-based ColorFunction:

Manipulate[
 ListLinePlot[data, Frame -> True, Axes -> False, 
  PlotRange -> {{n, n + win - 1}, All},
  ColorFunction -> (daycolour[Ceiling[#/24]] &), 
  ColorFunctionScaling -> False],
 {{n, 1, "First Hour"}, 1, 24*365 - win, 1},
 {{win, 168, "Window Length"}, 2, 336, 1}]