1
$\begingroup$

I'm using Windows 10 with MM12.2 PDF import does not include the standalone newlines anymore. This is a very annoying issue since we would like to extract paragraphs which is now not possible. We tried other PDF import methods: PDF2TEXT which successfully keeps the additional newlines when importing the .txt file as TXT. Export using Acrobat reader with "Save as text file". When importing the file as a text file the additional newlines are present.

We need to import thousands of PDF files and the new PDF import that came with version 12.2 is accepting a lot more PDF's than before but deletes the newlines the mark paragraphs. In fact TextCases relies on the "return/newlines" code to detect paragraphs (see:https://reference.wolfram.com/language/ref/textcontent/Paragraph.html) so combined with the new PDF importer doesn't really makes it useful.

Are there any known options to not delete the additional newlines?

An easy example is created below.

teststring = "sentence 1\nsentence 2\n\nsentence 3\n\n\nsentence 4" 
(*Create a string with additional newlines*)
Export["newlinepdf.pdf", teststring] (*export to PDF*)
wrongstring=Import["newlinepdf.pdf", "Plaintext"] (*Import again and notice that the additional newlines are missing. This behaviour is consistent with all PDF's we tried*)
Import["newlinepdf.pdf", "PlaintextLegacy"] (*The leagcy undocumented option works (but doubles the newlines)for this example but fails with other PDF's we tried (errors out)*)

Subscript[#, ToString@First@ToCharacterCode[#]] & /@ (teststring // Characters)

the teststring with charactercodes looks like this enter image description here

Subscript[#, ToString@First@ToCharacterCode[#]] & /@ (wrongstr // Characters)

The imported version with "Plaintext" option

enter image description here

Improve this question
$\endgroup$
3
  • $\begingroup$ To clarify, you're getting spaces, not new lines for "Plaintext" import? For me, on macos, I get the new lines; they're merged, e.g. "\n\n" is "\n" in the output, etc. But, I do get them. So, what OS are you using? $\endgroup$
    – rcollyer
    Commented Mar 15, 2021 at 19:38
  • $\begingroup$ I'm using windows 10 $\endgroup$
    – Lou
    Commented Mar 16, 2021 at 7:19
  • $\begingroup$ I updated the question. Thx for asking @rcolllyer ! $\endgroup$
    – Lou
    Commented Mar 16, 2021 at 7:55

1 Answer 1

Reset to default
2
$\begingroup$

The different text sections that get exported to the document really show up as separate text objects, not as a contiguous piece of text with literal newlines. Some software can say at best "there's more than one newline here", but if you look more closely that's about all they can say. I gave Acrobat Reader and a few other free PDF to text tools online with varying numbers of new lines, and no software was able to faithfully reproduce the exact number of newlines. On MacOS Acrobat Reader completely disregards all new lines, as does copying from Chrome or Preview.app. On Windows, Acrobat Reader only gives 1 or 2 newlines, despite adding 3+ newlines between the sentences. PlaintextLegacy used a different method of extracting text completely, and as you've noticed in general it was not nearly as reliable and is not guaranteed to be present in future versions of Wolfram Language. While I agree the newlines might be some nice info to have, it's not always possible to accurately preserve that data. The goal is to provide the plaintext itself, not necessarily spacing.

Improve this answer
$\endgroup$
4
  • $\begingroup$ @rcollyer mentioned he does get the newlines on MacOS with Import. So I feel it should be consistent between the platforms anyway. And indeed instead of 3 newlines just 2 of them is in any case enough to chase for Paragraphs. Thx for testing! I filed a case to Wolfram. $\endgroup$
    – Lou
    Commented Mar 16, 2021 at 16:23
  • $\begingroup$ @rcollyer mentioned that newlines are collapsed into a single newline. It is consistent between platforms, and not something which we can fix. You will get the same behavior in most text PDF software, including copying from browsers and native PDF apps. $\endgroup$
    – GenericAccountName
    Commented Mar 16, 2021 at 20:45
  • $\begingroup$ It seems that on the wolfram cloud it preserves at least one additional newline (so three newlines become 2) just like on MacOS. I wonder what the dependency is with Windows and Mathematica $\endgroup$
    – Lou
    Commented Mar 18, 2021 at 8:05
  • $\begingroup$ All versions of M in 12.2+ use the same text engine as chrome. You can test this by copying and pasting from chrome, you should always get the same results as the "Plaintext" element. $\endgroup$
    – GenericAccountName
    Commented Apr 7, 2021 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.