A fast bash pipe for TreeTagger

I have been working with the part-of-speech tagger developed at the IMS Stuttgart TreeTagger since my master thesis. It performs well on german texts as one could easily suppose, since it was one of its primary purposes. One major problem is that it’s poorly documented, so I would like to share the way that I found to pass things to TreeTagger through a pipe.

The first thing is that TreeTagger doesn’t take Unicode strings, as it dates back to the nineties. So you have to convert whatever you pass to ISO-8859-1, which the iconv software with the translit option set does very well. It means here “find an equivalent if the character cannot be exactly translated”.

Then you have to define the options that you want to use. I put the most frequent ones in the example.

Benefits

The advantage of a pipe is that you can clean the text while passing it to the tagger. Here is one way of doing it, by using the text editor sed to : 1. remove the trailing white lines 2. replace everything that’s more than one space by one space and 3. replacing spaces by new lines.

This way …

more ...

Resources and links of interest

Archive of links gathered during my PhD thesis:

  1. Linguistics and NLP
  2. Corpus Linguistics
  3. Perl
  4. LaTeX
  5. R
  6. PhD related
  7. Misc.

Update: Curated list of open-access/open-source/off-the-shelf resources and tools developed with a particular focus on German

1 – Linguistics and NLP

General Linguistics

Computational Linguistics

Online Articles and Conferences

Lists of CL Blogs

Resources for German

more ...