1
$\begingroup$

I made a variant problem to that given at Cleaning up a string list.

This time, the data are more ordered. Each game record has four elements:

  {away team's name, home team's name, the winner's all-caps name abbreviation and the 
    winner's score, the loser's all-caps name abbreviation and the loser's score}

as in:

  {"Assumption", "Grand Valley State", "GRV 43", " ASP 7"}

with the goal of reaching:

  res = {"Assumption","7", "Grand Valley State", "43"}

@lericr's solution for the previous problem:

  Riffle @@@  KeyValueMap[If[#1, Flatten[StringCases[#2, NumberString]],StringTrim[#2]] &] /@ KeySort /@ (GroupBy[#, StringContainsQ[NumberString]] & /@ lis)

reverses the score:

 res = {"Assumption", "43", "Grand Valley State", "7"}

Thanks for patience with this.

$\endgroup$
9
  • $\begingroup$ As I said in my answer to your previous version of this question, my solution assumes that the scores are in order respective to the names. I was under the impression that you were getting the data from somewhere and didn't have control over the format. If you have the ability to format (and order) the data how you want, then why is there any difficulty here at all? $\endgroup$
    – lericr
    Sep 19, 2023 at 18:23
  • 1
    $\begingroup$ I feel like we may be missing part of your problem. What is your actual difficulty? $\endgroup$
    – lericr
    Sep 19, 2023 at 18:23
  • 3
    $\begingroup$ And this new variant introduces new problems. How do we know abbreviations will be unique? How do we know how abbreviations are even created (e.g. why isn't "Grand Valley State" abbreviated as "GVS"?). We may need you to provide a mapping between full names and abbreviated names. And why would you sequence the scores based on winning/losing? Winner can be determined simply by higher score. $\endgroup$
    – lericr
    Sep 19, 2023 at 18:28
  • 2
    $\begingroup$ What control do you actually have over the form/structure of the data and what can you not control? If you give us the raw input that you have no control over, maybe we can help you figure out how best to transform it and then get your answer. $\endgroup$
    – lericr
    Sep 19, 2023 at 18:40
  • 1
    $\begingroup$ As I already mentioned, this latest incarnation of your data format is actually harder to deal with than what you showed in your previous question. So, your "cleaning" isn't really working. Give us the raw data format. $\endgroup$
    – lericr
    Sep 20, 2023 at 3:11

1 Answer 1

1
$\begingroup$

Okay, this will be tedious to set up, but once you get it working it should be easy after that (fingers crossed). You might be able to do further investigation around using ESPN's API that would simplify this.

It turns out that ESPN has an API that you can call directly to get structured data. For your purposes, you'd want the endpoint http://site.api.espn.com/apis/site/v2/sports/football/college-football/scoreboard. Unfortunately, I couldn't figure out how to get it to filter to these Div II/III games. It's easy to get the top 25 games, but that doesn't help you. However, you can get data for individual games if you have the game id. So, we'll do a three-step process where we first fetch game ids, use those ids to call the endpoint once per id, and then process the data we get for each response.

Step 1: scrape game ids

You'll need to get the url each time you want to do this. For the one you supplied in the comments, do this:

rawResponse = URLRead["https://www.espn.com/college-football/schedule/_/week/3/seasontype/2/group/35"];
gameIds = Union[StringCases[rawResponse["Body"], "gameId=" ~~ id : DigitCharacter .. :> id]];

For this case, I got 203 ids, which looks close (but I didn't check exactly).

Step 2: call the api

gameDataBaseUrl = "http://site.api.espn.com/apis/site/v2/sports/football/college-football/scoreboard/";
gamesData = URLExecute[gameDataBaseUrl <> #] & /@ gameIds;
(*This will take a few seconds*)

We should now have a bunch of structured data. If you look at the data, it'll be pretty confusing at first, just because there's so much stuff you don't care about. But you'll need to reverse engineer it like I did to understand the structure and how to get what you need. Start by just looking at a single game, like gamesData[[1]].

Step 3: process each game

Here's a function to transform the data for an individual game datum (might be a cleaner way to do this):

SummarizeGameData[gameData_] :=
  With[
    {competitors = SortBy[Lookup[Lookup[gameData, "competitions"], "competitors"][[1]], "homeAway"]},
    Flatten[MapAt[Lookup["shortDisplayName"], Lookup[competitors, {"team", "score"}], {All, 1}]]]

So, for example:

SummarizeGameData[gamesData[[2]]]
(* {Argonauts,10,Rattlers,31} *)

The data we got from the API was lists of rules. We can use Lookup on rule lists. If you looked at the structure of the data, you'll find that each game has a "competitions" property and each of those has a "competitors" property. The data you want is in that "competitors" property (which is weird, but I didn't design the API so don't blame me :) ). I sorted the "competitors" by the "homeAway" property, which results in the away team being first and home team being last, which is what I think you wanted.

Now we want a team name and that team's score. Inside of each of the "competitors" is a "team" and the name is buried in there. There are actually several names, and I chose "shortDisplayName". Also inside each competitor is a "score". So, I used Lookup to extract "team" and "score" and then applied another Lookup to the "team" data to get "shortDisplayName".

You can map this over gamesData:

SummarizeGameData /@ gamesData

If you do go down the rabbit hole of looking at the data structure, the Keys function will help you get a summary view at each level.

Keys[gamesData[[1]]]
(* {id,uid,date,name,shortName,season,week,competitions,links,status} *)

So, you could include the game's date or name. So, for example:

Lookup[gamesData[[1]], "name"]
(* Miles College Golden Bears at Arkansas-Pine Bluff Golden Lions *)

Lookup[gamesData[[1]], "shortName"]
(* MIL @ UAPB *)

Lookup[gamesData[[1]], "date"]
(* 2023-09-16T21:00Z *)

You could change the summarizing function like this:

SummarizeGameData[gameData_] :=
  With[
    {competitors = SortBy[Lookup[Lookup[gameData, "competitions"], "competitors"][[1]], "homeAway"]},
    Flatten[
      {Lookup[gameData, "date"], 
       Lookup[gameData, "shortName"], 
       MapAt[Lookup["shortDisplayName"], Lookup[competitors, {"team", "score"}], {All, 1}]}]]

And then you'd get:

SummarizeGameData[gamesData[[2]]]
(* {2023-09-16T22:00Z,WFLA @ FAMU,Argonauts,10,Rattlers,31} *)

You can also try the team abbreviations or team full names.

The api also has endpoints for team info, schedule info, and of course data for other sports.

Alternative

We could have tried to scrape all of the game data from the initial web page instead of going to the api. That would involve a lot of string parsing as well as logic to match the score to the away/home team as you wanted. The API approach seemed more robust to me and a lot simpler once you've figured out the schema. But of course, there's no guarantee that the API is stable.

Caveat

I couldn't find official documentation for the API, so I don't know what will happen for unusual circumstances or if there are better ways to get at the data. For example in your example there was a canceled game, but my game summary function will give you a 0-0 game result (I didn't investigate, but I would guess that there is some sort of property that explicitly says the game was canceled, and you could use that to trigger alternate behavior).

Resources

Here is the best (unofficial) documentation I could find:

https://gist.github.com/akeaswaran/b48b02f1c94f873c6655e7129910fc3b

$\endgroup$
2
  • $\begingroup$ at the risk of violating the comment guideline, lericr, thank you! You are an optimist regarding ESPN's API stability :) I have written code for your "alternative" suggestion, which did indeed involve lots of string parsing, and is also the subject of a quetion posted over at mathematica.stackexchange.com/questions/290560/… $\endgroup$
    – Suite401
    Sep 20, 2023 at 18:14
  • $\begingroup$ Regarding stability, I’d expect the api to be at least as stable as the human readable web page. $\endgroup$
    – lericr
    Sep 20, 2023 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.