Predicting when a SE user will reach a given rep

Question

TeX.SE meta proposed to predict when their number one user, egreg, will reach 654321 reputation points.

I made a prediction based on a linear interpolation of the 90 past days rep.

I thought it would be an interesting question to illustrate MMA statistical possibilities, especially if some statistics gurus could demonstrate the use of high-level MMA functions such as HypothesisTestData.

Here is the data of egreg's daily rep for the past 1460 days:

data = N@{247105, 247380, 247670, 247915, 248205, 248450, 248710, 
   249035, 249310, 249615, 249860, 250135, 250380, 250655, 250915, 
   251190, 251390, 251650, 251865, 252155, 252400, 252730, 252990, 
   253310, 253540, 253785, 254045, 254275, 254535, 254780, 255085, 
   255330, 255575, 255790, 256005, 256295, 256570, 256800, 257075, 
   257350, 257580, 257900, 258190, 258495, 258785, 259030, 259350, 
   259610, 259900, 260115, 260420, 260665, 260895, 261155, 261400, 
   261675, 261920, 262210, 262485, 262760, 263035, 263265, 263525, 
   263830, 264075, 264320, 264800, 265030, 265245, 265520, 265780, 
   266057, 266272, 266547, 266837, 267142, 267422, 267832, 268077, 
   268322, 268627, 268932, 269177, 269422, 269682, 269897, 270172, 
   270492, 270737, 270967, 271227, 271457, 271717, 271962, 272222, 
   272497, 272787, 273017, 273307, 273582, 273827, 274287, 274517, 
   274747, 275037, 275347, 275637, 275912, 276232, 276447, 276692, 
   276982, 277272, 277577, 277822, 278067, 278342, 278602, 278922, 
   279197, 279472, 279732, 280022, 280297, 280557, 280817, 281092, 
   281367, 281612, 281857, 282102, 282377, 282669, 282944, 283219, 
   283464, 283724, 283954, 284199, 284444, 284779, 285009, 285449, 
   285679, 285954, 286214, 286489, 286749, 287009, 287269, 287514, 
   287744, 287944, 288219, 288509, 288814, 289091, 289351, 289581, 
   289841, 290071, 290451, 290726, 290971, 291246, 291466, 291726, 
   292016, 292291, 292551, 292796, 293026, 293256, 293516, 293791, 
   294126, 294371, 294631, 294876, 295106, 295426, 295701, 295976, 
   296281, 296601, 296861, 297091, 297336, 297581, 297916, 298191, 
   298466, 298756, 299016, 299246, 299476, 299766, 300026, 300271, 
   300501, 300806, 301036, 301341, 301646, 301991, 302221, 302511, 
   302756, 303001, 303216, 303461, 303676, 303966, 304226, 304421, 
   304651, 304896, 305171, 305416, 305646, 305876, 306106, 306351, 
   306626, 306871, 307086, 307361, 307591, 307866, 307991, 308251, 
   308496, 308726, 309081, 309311, 309556, 309896, 310141, 310341, 
   310601, 310906, 311181, 311426, 311701, 312021, 312356, 312601, 
   312891, 313181, 313396, 313686, 313976, 314251, 314576, 314821, 
   315126, 315356, 315661, 315951, 316211, 316471, 316731, 317006, 
   317266, 317556, 317801, 318061, 318336, 318611, 318886, 319191, 
   319481, 319801, 320061, 320321, 320596, 320901, 321116, 321421, 
   321666, 321926, 322201, 322576, 322806, 323006, 323281, 323556, 
   323831, 324136, 324426, 324671, 324931, 325176, 325451, 325681, 
   325956, 326231, 326491, 326721, 327026, 327291, 327536, 327871, 
   328146, 328361, 328606, 328866, 329141, 329401, 329661, 329891, 
   330181, 330441, 330716, 331011, 331376, 331681, 331911, 332216, 
   332461, 332691, 332921, 333196, 333471, 333701, 333961, 334221, 
   334496, 334771, 335081, 335281, 335571, 335876, 336156, 336376, 
   336636, 336851, 337126, 337406, 337606, 337881, 338141, 338371, 
   338616, 338891, 339181, 339426, 339801, 340061, 340321, 340581, 
   340826, 341056, 341331, 341621, 341836, 342126, 342416, 342691, 
   342921, 343181, 343381, 343656, 343916, 344221, 344526, 344726, 
   345001, 345276, 345536, 345796, 346026, 346301, 346531, 346791, 
   347066, 347341, 347601, 347876, 348106, 348426, 348821, 349066, 
   349326, 349601, 349876, 350136, 350531, 350776, 351036, 351281, 
   351541, 351771, 352031, 352336, 352551, 352796, 353071, 353361, 
   353651, 353926, 354156, 354276, 354521, 354796, 355041, 355271, 
   355591, 355881, 356126, 356356, 356676, 356936, 357226, 357546, 
   357791, 358066, 358341, 358601, 358801, 359076, 359336, 359551, 
   359781, 360011, 360301, 360546, 360791, 361096, 361296, 361541, 
   361831, 362076, 362321, 362671, 362946, 363176, 363421, 363666, 
   363881, 364111, 364401, 364691, 364936, 365181, 365456, 365716, 
   366021, 366296, 366571, 366846, 367076, 367291, 367666, 367956, 
   368201, 368461, 368781, 369041, 369301, 369576, 369806, 370096, 
   370356, 370631, 370936, 371166, 371426, 371656, 371931, 372221, 
   372451, 372726, 373046, 373336, 373611, 373946, 374176, 374496, 
   374756, 375001, 375261, 375506, 375751, 376106, 376381, 376656, 
   376871, 377141, 377521, 377751, 377996, 378316, 378636, 378941, 
   379216, 379491, 379781, 380056, 380331, 380576, 380806, 381111, 
   381341, 381726, 381986, 382261, 382506, 382751, 383011, 383241, 
   383516, 383746, 384021, 384296, 384526, 384771, 385031, 385456, 
   385731, 385991, 386281, 386441, 386716, 387041, 387301, 387591, 
   387881, 388156, 388466, 388786, 389031, 389311, 389601, 389906, 
   390166, 390411, 390656, 390886, 391206, 391451, 391666, 391926, 
   392186, 392491, 392751, 392996, 393241, 393501, 393761, 393991, 
   394266, 394526, 394801, 395106, 395396, 395716, 395931, 396131, 
   396346, 396576, 396781, 396996, 397211, 397356, 397561, 397806, 
   398036, 398326, 398571, 398891, 399121, 399351, 399596, 399791, 
   400081, 400371, 400616, 400781, 401026, 401316, 401546, 401821, 
   402066, 402296, 402586, 402891, 403151, 403396, 403596, 403841, 
   404056, 404271, 404506, 404781, 405041, 405286, 405576, 405821, 
   406181, 406396, 406641, 406931, 407176, 407521, 407826, 408071, 
   408286, 408531, 408881, 409126, 409356, 409616, 409721, 409996, 
   410256, 410546, 410776, 411186, 411463, 411663, 411923, 412213, 
   412473, 412703, 412948, 413223, 413398, 413613, 413903, 414193, 
   414453, 414743, 415018, 415263, 415568, 415903, 416133, 416408, 
   416668, 416898, 417203, 417493, 417768, 418073, 418363, 418653, 
   418883, 419128, 419388, 419648, 419938, 420183, 420473, 420733, 
   420993, 421298, 421543, 421863, 422138, 422413, 422703, 422993, 
   423348, 423578, 423898, 424158, 424448, 424693, 424953, 425243, 
   425503, 425868, 426143, 426418, 426648, 426878, 427168, 427443, 
   427703, 427948, 428193, 428483, 428743, 428988, 429278, 429613, 
   429873, 430148, 430408, 430658, 430903, 431148, 431453, 431713, 
   431988, 432218, 432478, 432708, 432998, 433258, 433563, 433793, 
   434068, 434358, 434663, 434923, 435183, 435428, 435703, 435993, 
   436238, 436528, 436743, 437033, 437323, 437568, 437813, 438073, 
   438363, 438653, 438881, 439143, 439388, 439618, 439908, 440138, 
   440383, 440643, 440978, 441298, 441563, 441778, 442023, 442268, 
   442528, 442773, 443033, 443283, 443528, 443773, 444078, 444338, 
   444583, 444878, 445198, 445488, 445733, 446038, 446283, 446573, 
   446833, 447078, 447308, 447613, 447888, 448163, 448438, 448713, 
   448988, 449218, 449493, 449843, 450213, 450533, 450823, 451158, 
   451463, 451753, 452043, 452243, 452533, 452823, 453098, 453388, 
   453633, 453893, 454153, 454413, 454688, 454918, 455163, 455453, 
   455698, 455898, 456173, 456508, 456798, 457073, 457378, 457668, 
   457928, 458203, 458493, 458753, 459013, 459258, 459533, 459778, 
   460038, 460283, 460543, 460833, 461063, 461278, 461523, 461783, 
   462013, 462258, 462533, 462808, 463083, 463358, 463603, 463818, 
   464093, 464338, 464628, 464888, 465183, 465428, 465688, 465918, 
   466163, 466423, 466773, 467033, 467308, 467613, 467933, 468193, 
   468398, 468613, 468918, 469163, 469468, 469778, 470023, 470343, 
   470603, 470848, 471140, 471400, 471645, 471860, 472120, 472380, 
   472715, 472990, 473250, 473495, 473705, 473965, 474210, 474470, 
   474715, 475035, 475280, 475555, 475785, 476045, 476260, 476490, 
   476735, 476995, 477260, 477475, 477675, 477995, 478255, 478515, 
   478760, 479005, 479235, 479540, 479755, 480000, 480230, 480505, 
   480720, 481010, 481170, 481445, 481735, 482045, 482350, 482565, 
   482840, 483145, 483405, 483620, 483865, 484155, 484315, 484560, 
   484820, 485035, 485310, 485555, 485815, 485895, 486155, 486385, 
   486675, 486920, 487135, 487365, 487595, 487870, 488130, 488390, 
   488635, 488865, 489140, 489360, 489575, 489880, 490225, 490470, 
   490745, 491035, 491235, 491590, 491850, 492050, 492280, 492540, 
   492755, 492935, 493060, 493255, 493530, 493805, 494185, 494430, 
   494620, 494885, 495145, 495375, 495635, 495895, 496140, 496420, 
   496640, 496900, 497190, 497435, 497745, 497960, 498050, 498265, 
   498495, 498770, 499000, 499230, 499435, 499625, 499870, 500145, 
   500360, 500620, 500880, 501110, 501220, 501465, 501680, 501970, 
   502245, 502475, 502720, 502830, 502965, 503195, 503415, 503720, 
   504025, 504270, 504515, 504775, 504990, 505235, 505555, 505785, 
   506060, 506275, 506505, 506765, 507025, 507255, 507500, 507775, 
   508035, 508280, 508540, 508815, 509045, 509305, 509520, 509765, 
   510010, 510270, 510515, 510745, 510975, 511190, 511450, 511680, 
   511910, 512125, 512355, 512615, 512830, 513060, 513260, 513425, 
   513685, 513960, 514205, 514495, 514740, 515015, 515260, 515535, 
   515810, 516025, 516240, 516470, 516730, 516990, 517235, 517480, 
   517695, 517970, 518200, 518460, 518720, 518980, 519195, 519440, 
   519687, 519947, 520207, 520482, 520742, 521017, 521234, 521464, 
   521694, 521907, 522167, 522367, 522597, 522812, 523057, 523317, 
   523452, 523712, 523957, 524187, 524432, 524662, 524967, 525117, 
   525422, 525727, 525987, 526232, 526492, 526692, 526952, 527212, 
   527457, 527672, 527932, 528132, 528437, 528727, 528957, 529202, 
   529462, 529677, 530007, 530252, 530497, 530787, 531047, 531247, 
   531492, 531707, 531937, 532152, 532412, 532612, 532887, 533102, 
   533347, 533622, 533867, 534097, 534357, 534587, 534862, 535092, 
   535367, 535642, 535887, 536087, 536317, 536562, 536805, 537097, 
   537327, 537572, 537802, 538017, 538262, 538537, 538857, 539102, 
   539362, 539652, 539791, 540066, 540281, 540511, 540756, 540986, 
   541246, 541491, 541781, 542056, 542316, 542591, 542851, 543126, 
   543341, 543586, 543831, 544076, 544321, 544581, 544841, 545071, 
   545391, 545651, 545926, 546186, 546416, 546736, 546996, 547211, 
   547426, 547746, 547991, 548236, 548511, 548741, 548956, 549166, 
   549396, 549611, 549841, 550116, 550361, 550591, 550821, 551051, 
   551281, 551511, 551786, 552016, 552274, 552564, 552809, 553039, 
   553269, 553514, 553774, 554019, 554264, 554524, 554814, 555074, 
   555289, 555579, 555824, 556024, 556239, 556561, 556791, 557036, 
   557281, 557511, 557756, 558031, 558261, 558536, 558781, 559011, 
   559256, 559516, 559776, 560021, 560341, 560586, 560831, 561091, 
   561351, 561811, 562086, 562346, 562591, 562791, 563036, 563266, 
   563481, 563726, 563971, 564216, 564446, 564646, 564906, 565151, 
   565411, 565686, 565931, 566161, 566421, 566651, 566866, 567186, 
   567416, 567646, 567891, 568121, 568366, 568581, 568826, 569101, 
   569346, 569636, 569851, 570066, 570341, 570556, 570786, 571136, 
   571381, 571656, 571871, 572144, 572334, 572794, 573054, 573284, 
   573544, 573804, 573999, 574244, 574459, 574689, 574919, 575209, 
   575439, 575699, 575899, 576114, 576359, 576604, 576819, 577099, 
   577374, 577604, 577879, 578139, 578384, 578614, 578874, 579119, 
   579349, 579594, 579869, 580114, 580359, 580589, 580834, 581099, 
   581329, 581544, 581819, 582019, 582234, 582494, 582754, 583244, 
   583504, 583749, 583979, 584194, 584314, 584459, 584694, 584909, 
   585109, 585354, 585599, 585754, 585984, 586214, 586474, 586719, 
   586919, 587179, 587414, 587614, 587859, 588074, 588334, 588639, 
   588839, 588979, 589064, 589309, 589509, 589754, 589969, 590199, 
   590414, 590609, 590839, 591084, 591314, 591589, 591819, 592034, 
   592224, 592469, 592669, 592884, 593129, 593329, 593539, 593739, 
   593999, 594214, 594474, 594704, 594934, 595224, 595484, 595759, 
   596019, 596309, 596629, 596889, 597094, 597324, 597584, 597799, 
   598119, 598379, 598609, 598824, 599052, 599269, 599544, 599729, 
   600004, 600234, 600494, 600724, 600954, 601229, 601459, 601689, 
   601979, 602129, 602344, 602579, 602794, 603039, 603299, 603544, 
   603689, 603809, 604039, 604284, 604529, 604744, 604974, 605174, 
   605419, 605749, 605994, 606289, 606519, 606794, 607039, 607269, 
   607544, 607804, 608034, 608294, 608524, 608829, 609119, 609334, 
   609579, 609794, 610084, 610329, 610544, 610789, 611064, 611294, 
   611539, 611799, 612014, 612234, 612509, 612769, 613029, 613289, 
   613504, 613749, 613979, 614239, 614499, 614744, 614974, 615204, 
   615419, 615694, 615939, 616169, 616459, 616704, 616919, 617119, 
   617364, 617609, 617854, 618114, 618344, 618589, 618849, 619094, 
   619249, 619434, 619644, 619889, 620149, 620379, 620574, 620834, 
   621064, 621279, 621584, 621829, 622032}

Answer 1

For what follows, rep contains the reputation observations over the given sample (1.460 days).

Fitting a linear trend on the data eg lmf=LiearModelFit[rep,t,t] produces the following fitted function (evaluating lmf["Function"][t] produces):

250144. + 258.184 t

t here are 'days' but that is not made explicit in the estimation; plugging in t=1460 returns 627092.; that is the estimated reputation for the last point in the sample (the actual value is 622032).

Also, according to this linear model, the reputation target trep=654321 will be obtained on day 1566 or 106 days after the end of our sample. We obtain the point estimate of a reputation value that is equal to or greater to the target figure using the follow snippet of code:

Reap[
  Scan[
    If[lmf[#] >= trep, Sow[#]; Return[]] &, 1460 + Range[365]]]  

According to this, the target figure will be obtained on

DatePlus[{2018, 1, 5}, {106, "Day"}]

{2018, 4, 21}

or as of late April, 2018.

Now, all the above are great and all, with a minor glitch; take a look at the residuals from the estimated model:

This is what a major autocorrelation infestation looks like. The presence of autocorrelation in the residuals can be interpreted in different ways; for the present case we'll go with model mis-specification ie there is a systematic part in the data that (systematically?) eludes our (simple, linear trend) model.

The linear model has persistent positive residual autocorrelation.

In order to find a better specified model, we'll turn to Mathematica's TimeSeriesModelFit. For the rest of this segment, tsrep stands for

d1 = {2018, 1, 5};
delta = 1460;
d0 = DatePlus[d1, {-delta + 1, "Day"}]

tsrep = TimeSeries[rep, {d0, Automatic, "Day"}, 
  MetaInformation -> {"User" -> "TeX.SE@egreg", "Units" -> "Reputation"}]

the TimeSeries representation of rep.

This is what the reputation data look like

xlbl = Row[{delta, " ", "days of data"}];
tspec = {"Year", "/", "Month", "/","Day"};
from = DateString[tsrep["FirstTime"], tspec];
to = DateString[tsrep["LastTime"], tspec];
txt = "data provided by user Mma.SE@anderstood";

DateListPlot[tsrep , 
  Filling -> Axis, 
  FrameLabel -> {
    {tsrep["Units"], Automatic}, 
    {xlbl, Row[{from, "-", to}]}
   }, 
  PlotLegends -> tsrep["User"], 
  Epilog -> {Text[Style[txt, Darker@Red, Bold, Italic], Scaled[{0.5, 0.1}]]}]

The data display a definite trend-which we'll assume is deterministic; this means that the reputation series as it stands is a non-stationary series. This observation is confirmed by the outcome of performing appropriate unit-root tests:

headers = Style[#, Bold] & /@ {"Lags", "Test", "Test statistic", 
  "P-value", "Test conclusion"};
Prepend[Prepend[
  UnitRootTest[rep, 
    {"Drift", #}, 
    {"AutomaticTest", "TestStatistic", "PValue", "ShortTestConclusion"}
   ], #] & /@ {1, 2, 7, 15, 30}, headers] // Grid

The null of the presence of a unit root cannot be rejected for various specifications of the test.

We are going to split our data into two parts: the first one we'll use for estimation purposes and the later part will be set aside for testing purposes. We choose to split the data at {2017, 10, 8} (last 90 days of data).

delta = 90;
d2 = DatePlus[d1, {-delta + 1, "Day"}]
excludeLast90dDays = TimeSeriesWindow[tsrep, {Automatic, d2}];
last90Days = TimeSeriesWindow[tsrep, {d2, Automatic}];

We let Mathematica select an appropriate model for the first part of our data

mx90 = TimeSeriesModelFit[excludeLast90dDays];

The selected model is an ARIMAProcess[-0.034365, {-0.0386807, -0.0263645, 0.0195861}, 2, {-0.964332}, 1528.1]. The residuals obtained from this model as well as their scatter plots at various lags are reported below.

The ACF's, PACF's and Ljung-Box don't seem terribly bad also.

This time, the model predicts that trep will be obtained

Reap[Scan[
 If[mx90[#] >= trep, Sow[#]; Return[]] &, 
  DateRange[d2, DatePlus[d1, {365, "Day"}], "Day"]]
]

on

{2018, 5, 25}.

The predicted date along with appropriate 95% bounds according to the ARIMA model is presented below

---

code

(* used to obtain lags of residuals *)
pair[list_, lag_] := {Drop[list, -lag], Drop[list, lag]}

Clear[u, t]
(* used to scatter-plot residuals *)
plot[list_, lag_] := ListPlot[
  pair[list, lag] // Transpose, 
  Frame -> True, 
  FrameLabel -> {
    {Subscript[u, t], Automatic}, 
    {Subscript[u, t - lag],Row[{"lag = ", lag, " ", "day(s)"}]}
   }, 
  Axes -> False, 
  ImageSize -> Small, 
  RotateLabel -> False]

Answer 2

Data in the format {date, reputation}...:

timeddata = Transpose[{Range[Length[data]], data}];

A quadratic fit:

mymodel = Fit[timeddata, {1, x, x^2}, x]

(* 245614. + 276.773 x - 0.0127237 x^2 *)

Solve for criterion:

Solve[245614.0277534589` + 276.7728073237194` x - 
   0.012723685322656041` x^2 == 654321, x]

(* {{x -> 1593.41}, {x -> 20159.2}} *)

The first solution is the correct one, of course: 1593 days from the beginning of the data collection.

A graph:

Show[
 ListPlot[hhh, PlotStyle -> Red, 
  PlotRange -> {{0, 2000}, {0, 700000}}],
 Plot[{654321, 
   245614.0277534589` + 276.7728073237194` x - 
    0.012723685322656041` x^2}, {x, 1, 2000}],
 Epilog -> {Dashed, Line[{{1593, 0}, {1593, 654321}}]}]

You can add cubic or quartic terms, but it doesn't change the predicted date enough to matter--the cubit term in the fit is less than $10^{-5}$.