How to work around HTML not importing full HTML as XMLObject

Question

I'm trying to parse some data out of the HTML for a form. Here's a simple one

CloudDeploy[
 FormFunction[
  {"Name" -> "String", "Email" -> "EmailAddress"},
  TemplateApply["Thanks ``! Your submission went through.", {#Name}] &
  ],
 "testForm",
 Permissions -> "Public"
 ]

It's easy enough to get the raw HTML body

body = URLRead[CloudObject["testForm"]]["Body"]

and I see a large <script> in the header + some stuff in the body.

But if I use Import that all goes away

ImportString[body, {"HTML", "XMLObject"}]

XMLObject["Document"][{XMLObject["Declaration"]["Version" -> "1.0", 
   "Standalone" -> "yes"]}, 
 XMLElement[
  "html", {"lang" -> "en", {"http://www.w3.org/2000/xmlns/", "xmlns"} -> 
    "http://www.w3.org/1999/xhtml"}, {XMLElement[
    "head", {}, {XMLElement["title", {}, {"
      Wolfram
    "}], XMLElement["meta", {"charset" -> "utf-8"}, {}], 
     ...,
     XMLElement[
      "script", {"defer" -> "defer", "type" -> "text/javascript", 
       "src" -> 
        "/dist/dyn/vendors~Dialogs~account~authLanding~dashboard~deployedView~\
directoryPlaceholderLanding~error401Permi~17e11331-4e693e624e6d42331857.js"}, \
{}], XMLElement[
      "script", {"defer" -> "defer", "type" -> "text/javascript", 
       "src" -> "/dist/dyn/formView-4db6f811488735b8a396.js"}, {}], 
     XMLElement["script", {}, {"
if(typeof LAZY_RESOURCES === 'undefined'){LAZY_RESOURCES={}}
"}]}]}], {}]

I tracked the issue down to System`Convert`HTMLImportDump`FixFile but I'm not sure how I should best patch this since the damage happens on the Java side (the function uses JLink)

Answer 1

Here is a quick workaround using "Python" with "beautifulsoup4", a third-party module to format the HTML text.

1. First install "beautifulsoup4"/"bs4" (pypi)

2. create a function to use the module

prettify = ExternalFunction["Python", "
from bs4 import BeautifulSoup as bs

def temp(text):
    return bs(text).prettify()
"]

apply it to strings:

ImportString[text, {"HTML", "XMLObject"}] // Depth
(* Out: 9 *)

ImportString[prettify @ text, {"HTML", "XMLObject"}] // Depth
(* Out: 27 *)