7
$\begingroup$

TeX.SE meta proposed to predict when their number one user, egreg, will reach 654321 reputation points.

I made a prediction based on a linear interpolation of the 90 past days rep.

I thought it would be an interesting question to illustrate MMA statistical possibilities, especially if some statistics gurus could demonstrate the use of high-level MMA functions such as HypothesisTestData.

Here is the data of egreg's daily rep for the past 1460 days:

data = N@{247105, 247380, 247670, 247915, 248205, 248450, 248710, 
   249035, 249310, 249615, 249860, 250135, 250380, 250655, 250915, 
   251190, 251390, 251650, 251865, 252155, 252400, 252730, 252990, 
   253310, 253540, 253785, 254045, 254275, 254535, 254780, 255085, 
   255330, 255575, 255790, 256005, 256295, 256570, 256800, 257075, 
   257350, 257580, 257900, 258190, 258495, 258785, 259030, 259350, 
   259610, 259900, 260115, 260420, 260665, 260895, 261155, 261400, 
   261675, 261920, 262210, 262485, 262760, 263035, 263265, 263525, 
   263830, 264075, 264320, 264800, 265030, 265245, 265520, 265780, 
   266057, 266272, 266547, 266837, 267142, 267422, 267832, 268077, 
   268322, 268627, 268932, 269177, 269422, 269682, 269897, 270172, 
   270492, 270737, 270967, 271227, 271457, 271717, 271962, 272222, 
   272497, 272787, 273017, 273307, 273582, 273827, 274287, 274517, 
   274747, 275037, 275347, 275637, 275912, 276232, 276447, 276692, 
   276982, 277272, 277577, 277822, 278067, 278342, 278602, 278922, 
   279197, 279472, 279732, 280022, 280297, 280557, 280817, 281092, 
   281367, 281612, 281857, 282102, 282377, 282669, 282944, 283219, 
   283464, 283724, 283954, 284199, 284444, 284779, 285009, 285449, 
   285679, 285954, 286214, 286489, 286749, 287009, 287269, 287514, 
   287744, 287944, 288219, 288509, 288814, 289091, 289351, 289581, 
   289841, 290071, 290451, 290726, 290971, 291246, 291466, 291726, 
   292016, 292291, 292551, 292796, 293026, 293256, 293516, 293791, 
   294126, 294371, 294631, 294876, 295106, 295426, 295701, 295976, 
   296281, 296601, 296861, 297091, 297336, 297581, 297916, 298191, 
   298466, 298756, 299016, 299246, 299476, 299766, 300026, 300271, 
   300501, 300806, 301036, 301341, 301646, 301991, 302221, 302511, 
   302756, 303001, 303216, 303461, 303676, 303966, 304226, 304421, 
   304651, 304896, 305171, 305416, 305646, 305876, 306106, 306351, 
   306626, 306871, 307086, 307361, 307591, 307866, 307991, 308251, 
   308496, 308726, 309081, 309311, 309556, 309896, 310141, 310341, 
   310601, 310906, 311181, 311426, 311701, 312021, 312356, 312601, 
   312891, 313181, 313396, 313686, 313976, 314251, 314576, 314821, 
   315126, 315356, 315661, 315951, 316211, 316471, 316731, 317006, 
   317266, 317556, 317801, 318061, 318336, 318611, 318886, 319191, 
   319481, 319801, 320061, 320321, 320596, 320901, 321116, 321421, 
   321666, 321926, 322201, 322576, 322806, 323006, 323281, 323556, 
   323831, 324136, 324426, 324671, 324931, 325176, 325451, 325681, 
   325956, 326231, 326491, 326721, 327026, 327291, 327536, 327871, 
   328146, 328361, 328606, 328866, 329141, 329401, 329661, 329891, 
   330181, 330441, 330716, 331011, 331376, 331681, 331911, 332216, 
   332461, 332691, 332921, 333196, 333471, 333701, 333961, 334221, 
   334496, 334771, 335081, 335281, 335571, 335876, 336156, 336376, 
   336636, 336851, 337126, 337406, 337606, 337881, 338141, 338371, 
   338616, 338891, 339181, 339426, 339801, 340061, 340321, 340581, 
   340826, 341056, 341331, 341621, 341836, 342126, 342416, 342691, 
   342921, 343181, 343381, 343656, 343916, 344221, 344526, 344726, 
   345001, 345276, 345536, 345796, 346026, 346301, 346531, 346791, 
   347066, 347341, 347601, 347876, 348106, 348426, 348821, 349066, 
   349326, 349601, 349876, 350136, 350531, 350776, 351036, 351281, 
   351541, 351771, 352031, 352336, 352551, 352796, 353071, 353361, 
   353651, 353926, 354156, 354276, 354521, 354796, 355041, 355271, 
   355591, 355881, 356126, 356356, 356676, 356936, 357226, 357546, 
   357791, 358066, 358341, 358601, 358801, 359076, 359336, 359551, 
   359781, 360011, 360301, 360546, 360791, 361096, 361296, 361541, 
   361831, 362076, 362321, 362671, 362946, 363176, 363421, 363666, 
   363881, 364111, 364401, 364691, 364936, 365181, 365456, 365716, 
   366021, 366296, 366571, 366846, 367076, 367291, 367666, 367956, 
   368201, 368461, 368781, 369041, 369301, 369576, 369806, 370096, 
   370356, 370631, 370936, 371166, 371426, 371656, 371931, 372221, 
   372451, 372726, 373046, 373336, 373611, 373946, 374176, 374496, 
   374756, 375001, 375261, 375506, 375751, 376106, 376381, 376656, 
   376871, 377141, 377521, 377751, 377996, 378316, 378636, 378941, 
   379216, 379491, 379781, 380056, 380331, 380576, 380806, 381111, 
   381341, 381726, 381986, 382261, 382506, 382751, 383011, 383241, 
   383516, 383746, 384021, 384296, 384526, 384771, 385031, 385456, 
   385731, 385991, 386281, 386441, 386716, 387041, 387301, 387591, 
   387881, 388156, 388466, 388786, 389031, 389311, 389601, 389906, 
   390166, 390411, 390656, 390886, 391206, 391451, 391666, 391926, 
   392186, 392491, 392751, 392996, 393241, 393501, 393761, 393991, 
   394266, 394526, 394801, 395106, 395396, 395716, 395931, 396131, 
   396346, 396576, 396781, 396996, 397211, 397356, 397561, 397806, 
   398036, 398326, 398571, 398891, 399121, 399351, 399596, 399791, 
   400081, 400371, 400616, 400781, 401026, 401316, 401546, 401821, 
   402066, 402296, 402586, 402891, 403151, 403396, 403596, 403841, 
   404056, 404271, 404506, 404781, 405041, 405286, 405576, 405821, 
   406181, 406396, 406641, 406931, 407176, 407521, 407826, 408071, 
   408286, 408531, 408881, 409126, 409356, 409616, 409721, 409996, 
   410256, 410546, 410776, 411186, 411463, 411663, 411923, 412213, 
   412473, 412703, 412948, 413223, 413398, 413613, 413903, 414193, 
   414453, 414743, 415018, 415263, 415568, 415903, 416133, 416408, 
   416668, 416898, 417203, 417493, 417768, 418073, 418363, 418653, 
   418883, 419128, 419388, 419648, 419938, 420183, 420473, 420733, 
   420993, 421298, 421543, 421863, 422138, 422413, 422703, 422993, 
   423348, 423578, 423898, 424158, 424448, 424693, 424953, 425243, 
   425503, 425868, 426143, 426418, 426648, 426878, 427168, 427443, 
   427703, 427948, 428193, 428483, 428743, 428988, 429278, 429613, 
   429873, 430148, 430408, 430658, 430903, 431148, 431453, 431713, 
   431988, 432218, 432478, 432708, 432998, 433258, 433563, 433793, 
   434068, 434358, 434663, 434923, 435183, 435428, 435703, 435993, 
   436238, 436528, 436743, 437033, 437323, 437568, 437813, 438073, 
   438363, 438653, 438881, 439143, 439388, 439618, 439908, 440138, 
   440383, 440643, 440978, 441298, 441563, 441778, 442023, 442268, 
   442528, 442773, 443033, 443283, 443528, 443773, 444078, 444338, 
   444583, 444878, 445198, 445488, 445733, 446038, 446283, 446573, 
   446833, 447078, 447308, 447613, 447888, 448163, 448438, 448713, 
   448988, 449218, 449493, 449843, 450213, 450533, 450823, 451158, 
   451463, 451753, 452043, 452243, 452533, 452823, 453098, 453388, 
   453633, 453893, 454153, 454413, 454688, 454918, 455163, 455453, 
   455698, 455898, 456173, 456508, 456798, 457073, 457378, 457668, 
   457928, 458203, 458493, 458753, 459013, 459258, 459533, 459778, 
   460038, 460283, 460543, 460833, 461063, 461278, 461523, 461783, 
   462013, 462258, 462533, 462808, 463083, 463358, 463603, 463818, 
   464093, 464338, 464628, 464888, 465183, 465428, 465688, 465918, 
   466163, 466423, 466773, 467033, 467308, 467613, 467933, 468193, 
   468398, 468613, 468918, 469163, 469468, 469778, 470023, 470343, 
   470603, 470848, 471140, 471400, 471645, 471860, 472120, 472380, 
   472715, 472990, 473250, 473495, 473705, 473965, 474210, 474470, 
   474715, 475035, 475280, 475555, 475785, 476045, 476260, 476490, 
   476735, 476995, 477260, 477475, 477675, 477995, 478255, 478515, 
   478760, 479005, 479235, 479540, 479755, 480000, 480230, 480505, 
   480720, 481010, 481170, 481445, 481735, 482045, 482350, 482565, 
   482840, 483145, 483405, 483620, 483865, 484155, 484315, 484560, 
   484820, 485035, 485310, 485555, 485815, 485895, 486155, 486385, 
   486675, 486920, 487135, 487365, 487595, 487870, 488130, 488390, 
   488635, 488865, 489140, 489360, 489575, 489880, 490225, 490470, 
   490745, 491035, 491235, 491590, 491850, 492050, 492280, 492540, 
   492755, 492935, 493060, 493255, 493530, 493805, 494185, 494430, 
   494620, 494885, 495145, 495375, 495635, 495895, 496140, 496420, 
   496640, 496900, 497190, 497435, 497745, 497960, 498050, 498265, 
   498495, 498770, 499000, 499230, 499435, 499625, 499870, 500145, 
   500360, 500620, 500880, 501110, 501220, 501465, 501680, 501970, 
   502245, 502475, 502720, 502830, 502965, 503195, 503415, 503720, 
   504025, 504270, 504515, 504775, 504990, 505235, 505555, 505785, 
   506060, 506275, 506505, 506765, 507025, 507255, 507500, 507775, 
   508035, 508280, 508540, 508815, 509045, 509305, 509520, 509765, 
   510010, 510270, 510515, 510745, 510975, 511190, 511450, 511680, 
   511910, 512125, 512355, 512615, 512830, 513060, 513260, 513425, 
   513685, 513960, 514205, 514495, 514740, 515015, 515260, 515535, 
   515810, 516025, 516240, 516470, 516730, 516990, 517235, 517480, 
   517695, 517970, 518200, 518460, 518720, 518980, 519195, 519440, 
   519687, 519947, 520207, 520482, 520742, 521017, 521234, 521464, 
   521694, 521907, 522167, 522367, 522597, 522812, 523057, 523317, 
   523452, 523712, 523957, 524187, 524432, 524662, 524967, 525117, 
   525422, 525727, 525987, 526232, 526492, 526692, 526952, 527212, 
   527457, 527672, 527932, 528132, 528437, 528727, 528957, 529202, 
   529462, 529677, 530007, 530252, 530497, 530787, 531047, 531247, 
   531492, 531707, 531937, 532152, 532412, 532612, 532887, 533102, 
   533347, 533622, 533867, 534097, 534357, 534587, 534862, 535092, 
   535367, 535642, 535887, 536087, 536317, 536562, 536805, 537097, 
   537327, 537572, 537802, 538017, 538262, 538537, 538857, 539102, 
   539362, 539652, 539791, 540066, 540281, 540511, 540756, 540986, 
   541246, 541491, 541781, 542056, 542316, 542591, 542851, 543126, 
   543341, 543586, 543831, 544076, 544321, 544581, 544841, 545071, 
   545391, 545651, 545926, 546186, 546416, 546736, 546996, 547211, 
   547426, 547746, 547991, 548236, 548511, 548741, 548956, 549166, 
   549396, 549611, 549841, 550116, 550361, 550591, 550821, 551051, 
   551281, 551511, 551786, 552016, 552274, 552564, 552809, 553039, 
   553269, 553514, 553774, 554019, 554264, 554524, 554814, 555074, 
   555289, 555579, 555824, 556024, 556239, 556561, 556791, 557036, 
   557281, 557511, 557756, 558031, 558261, 558536, 558781, 559011, 
   559256, 559516, 559776, 560021, 560341, 560586, 560831, 561091, 
   561351, 561811, 562086, 562346, 562591, 562791, 563036, 563266, 
   563481, 563726, 563971, 564216, 564446, 564646, 564906, 565151, 
   565411, 565686, 565931, 566161, 566421, 566651, 566866, 567186, 
   567416, 567646, 567891, 568121, 568366, 568581, 568826, 569101, 
   569346, 569636, 569851, 570066, 570341, 570556, 570786, 571136, 
   571381, 571656, 571871, 572144, 572334, 572794, 573054, 573284, 
   573544, 573804, 573999, 574244, 574459, 574689, 574919, 575209, 
   575439, 575699, 575899, 576114, 576359, 576604, 576819, 577099, 
   577374, 577604, 577879, 578139, 578384, 578614, 578874, 579119, 
   579349, 579594, 579869, 580114, 580359, 580589, 580834, 581099, 
   581329, 581544, 581819, 582019, 582234, 582494, 582754, 583244, 
   583504, 583749, 583979, 584194, 584314, 584459, 584694, 584909, 
   585109, 585354, 585599, 585754, 585984, 586214, 586474, 586719, 
   586919, 587179, 587414, 587614, 587859, 588074, 588334, 588639, 
   588839, 588979, 589064, 589309, 589509, 589754, 589969, 590199, 
   590414, 590609, 590839, 591084, 591314, 591589, 591819, 592034, 
   592224, 592469, 592669, 592884, 593129, 593329, 593539, 593739, 
   593999, 594214, 594474, 594704, 594934, 595224, 595484, 595759, 
   596019, 596309, 596629, 596889, 597094, 597324, 597584, 597799, 
   598119, 598379, 598609, 598824, 599052, 599269, 599544, 599729, 
   600004, 600234, 600494, 600724, 600954, 601229, 601459, 601689, 
   601979, 602129, 602344, 602579, 602794, 603039, 603299, 603544, 
   603689, 603809, 604039, 604284, 604529, 604744, 604974, 605174, 
   605419, 605749, 605994, 606289, 606519, 606794, 607039, 607269, 
   607544, 607804, 608034, 608294, 608524, 608829, 609119, 609334, 
   609579, 609794, 610084, 610329, 610544, 610789, 611064, 611294, 
   611539, 611799, 612014, 612234, 612509, 612769, 613029, 613289, 
   613504, 613749, 613979, 614239, 614499, 614744, 614974, 615204, 
   615419, 615694, 615939, 616169, 616459, 616704, 616919, 617119, 
   617364, 617609, 617854, 618114, 618344, 618589, 618849, 619094, 
   619249, 619434, 619644, 619889, 620149, 620379, 620574, 620834, 
   621064, 621279, 621584, 621829, 622032}
Improve this question
$\endgroup$
  • $\begingroup$ My contribution there. $\endgroup$ – corey979 Jan 5 '18 at 19:13
  • $\begingroup$ @corey979 Too bad you break rule number 7 :P. Seriously, there is a big discrepancy between our answers. One of use must have made a mistake... $\endgroup$ – anderstood Jan 5 '18 at 19:26
  • $\begingroup$ Nevermind rule 7 :p You can run my codes by yourself and we'll see what's the result (on every OS and MMA ver FindFormula gives different outputs). But you used the last 90 values, while I took all 1460, so no wonder there's a difference. $\endgroup$ – corey979 Jan 5 '18 at 19:35
  • $\begingroup$ @corey979 Actually FindFormula gives me always 0 for $x>=1460$ so I can't reproduce your result. However I used all the data points in my approach and I found only 5 seconds difference with your result. The data is not as linear as it seems! $\endgroup$ – anderstood Jan 5 '18 at 19:49
  • $\begingroup$ @corey979 On my PC all the answers I get are like your first (with piecewise). Checked for 20 formulas. $\endgroup$ – anderstood Jan 5 '18 at 19:55
6
$\begingroup$

For what follows, rep contains the reputation observations over the given sample (1.460 days).

Fitting a linear trend on the data eg lmf=LiearModelFit[rep,t,t] produces the following fitted function (evaluating lmf["Function"][t] produces):

250144. + 258.184 t

t here are 'days' but that is not made explicit in the estimation; plugging in t=1460 returns 627092.; that is the estimated reputation for the last point in the sample (the actual value is 622032).

Also, according to this linear model, the reputation target trep=654321 will be obtained on day 1566 or 106 days after the end of our sample. We obtain the point estimate of a reputation value that is equal to or greater to the target figure using the follow snippet of code:

Reap[
  Scan[
    If[lmf[#] >= trep, Sow[#]; Return[]] &, 1460 + Range[365]]]

According to this, the target figure will be obtained on 

DatePlus[{2018, 1, 5}, {106, "Day"}]

{2018, 4, 21}

or as of late April, 2018.

Now, all the above are great and all, with a minor glitch; take a look at the residuals from the estimated model:

enter image description here

This is what a major autocorrelation infestation looks like. The presence of autocorrelation in the residuals can be interpreted in different ways; for the present case we'll go with model mis-specification ie there is a systematic part in the data that (systematically?) eludes our (simple, linear trend) model.

The linear model has persistent positive residual autocorrelation.

enter image description here

In order to find a better specified model, we'll turn to Mathematica's TimeSeriesModelFit. For the rest of this segment, tsrep stands for

d1 = {2018, 1, 5};
delta = 1460;
d0 = DatePlus[d1, {-delta + 1, "Day"}]

tsrep = TimeSeries[rep, {d0, Automatic, "Day"}, 
  MetaInformation -> {"User" -> "TeX.SE@egreg", "Units" -> "Reputation"}]

the TimeSeries representation of rep.

This is what the reputation data look like

xlbl = Row[{delta, " ", "days of data"}];
tspec = {"Year", "/", "Month", "/","Day"};
from = DateString[tsrep["FirstTime"], tspec];
to = DateString[tsrep["LastTime"], tspec];
txt = "data provided by user Mma.SE@anderstood";

DateListPlot[tsrep , 
  Filling -> Axis, 
  FrameLabel -> {
    {tsrep["Units"], Automatic}, 
    {xlbl, Row[{from, "-", to}]}
   }, 
  PlotLegends -> tsrep["User"], 
  Epilog -> {Text[Style[txt, Darker@Red, Bold, Italic], Scaled[{0.5, 0.1}]]}]

enter image description here

The data display a definite trend-which we'll assume is deterministic; this means that the reputation series as it stands is a non-stationary series. This observation is confirmed by the outcome of performing appropriate unit-root tests:

headers = Style[#, Bold] & /@ {"Lags", "Test", "Test statistic", 
  "P-value", "Test conclusion"};
Prepend[Prepend[
  UnitRootTest[rep, 
    {"Drift", #}, 
    {"AutomaticTest", "TestStatistic", "PValue", "ShortTestConclusion"}
   ], #] & /@ {1, 2, 7, 15, 30}, headers] // Grid

enter image description here

The null of the presence of a unit root cannot be rejected for various specifications of the test.

We are going to split our data into two parts: the first one we'll use for estimation purposes and the later part will be set aside for testing purposes. We choose to split the data at {2017, 10, 8} (last 90 days of data).

delta = 90;
d2 = DatePlus[d1, {-delta + 1, "Day"}]
excludeLast90dDays = TimeSeriesWindow[tsrep, {Automatic, d2}];
last90Days = TimeSeriesWindow[tsrep, {d2, Automatic}];

We let Mathematica select an appropriate model for the first part of our data

mx90 = TimeSeriesModelFit[excludeLast90dDays];

The selected model is an ARIMAProcess[-0.034365, {-0.0386807, -0.0263645, 0.0195861}, 2, {-0.964332}, 1528.1]. The residuals obtained from this model as well as their scatter plots at various lags are reported below.

enter image description here

enter image description here

The ACF's, PACF's and Ljung-Box don't seem terribly bad also.

enter image description here

This time, the model predicts that trep will be obtained 

Reap[Scan[
 If[mx90[#] >= trep, Sow[#]; Return[]] &, 
  DateRange[d2, DatePlus[d1, {365, "Day"}], "Day"]]
]

on

{2018, 5, 25}.

The predicted date along with appropriate 95% bounds according to the ARIMA model is presented below

enter image description here

code

(* used to obtain lags of residuals *)
pair[list_, lag_] := {Drop[list, -lag], Drop[list, lag]}

Clear[u, t]
(* used to scatter-plot residuals *)
plot[list_, lag_] := ListPlot[
  pair[list, lag] // Transpose, 
  Frame -> True, 
  FrameLabel -> {
    {Subscript[u, t], Automatic}, 
    {Subscript[u, t - lag],Row[{"lag = ", lag, " ", "day(s)"}]}
   }, 
  Axes -> False, 
  ImageSize -> Small, 
  RotateLabel -> False]
Improve this answer
$\endgroup$
  • $\begingroup$ Exactly the kind of answers I was hoping for! You did not use the last90days, for testing purposes right? $\endgroup$ – anderstood Jan 6 '18 at 16:47
  • $\begingroup$ Also, are they objective reasons why you split at 90 days, and not another date? $\endgroup$ – anderstood Jan 6 '18 at 18:26
  • $\begingroup$ I was planning to test multiple models but the answer started growing a little too much so I figured I stopped there; no, there's no objective reason that I know of, to split at 90; it just seemed reasonable to keep the most recent data for testing; the process dynamics are already evident; just to be clear, splitting the data into two sets is common practice-choosing the 90 day mark is arbitrary $\endgroup$ – user42582 Jan 6 '18 at 20:12
3
$\begingroup$

Data in the format {date, reputation}...:

timeddata = Transpose[{Range[Length[data]], data}];

A quadratic fit:

mymodel = Fit[timeddata, {1, x, x^2}, x]

(* 245614. + 276.773 x - 0.0127237 x^2 *)

Solve for criterion:

Solve[245614.0277534589` + 276.7728073237194` x - 
   0.012723685322656041` x^2 == 654321, x]

(* {{x -> 1593.41}, {x -> 20159.2}} *)

The first solution is the correct one, of course: 1593 days from the beginning of the data collection.

A graph:

Show[
 ListPlot[hhh, PlotStyle -> Red, 
  PlotRange -> {{0, 2000}, {0, 700000}}],
 Plot[{654321, 
   245614.0277534589` + 276.7728073237194` x - 
    0.012723685322656041` x^2}, {x, 1, 2000}],
 Epilog -> {Dashed, Line[{{1593, 0}, {1593, 654321}}]}]

enter image description here

You can add cubic or quartic terms, but it doesn't change the predicted date enough to matter--the cubit term in the fit is less than $10^{-5}$.

Improve this answer
$\endgroup$
  • 1
    $\begingroup$ For comparison with other answers, that gives DatePlus[{2018, 1, 5}, {1593 - Length@data, "Day"}] ie May 18th, 2018. $\endgroup$ – anderstood Jan 6 '18 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.