Evaluating the quality of a Language Model should be done by comparisons based on the right metrics for your model type.

Training Language Models should start with a simple baseline and be improved with more complex techniques.

Identifying the language of a text is often done before you select the right language model.

Extracting text from a file should be accompanied with the extraction of meta-information.

Pre-processing text with the goal to increase the quality of subsequent NLP tasks.

Finding texts that are exactly the same or show a high similarity. Similarity can be measured on lexicality or semantic meaning from embeddings.

Measuring the readability of a text by looking at the keyword density, syllable count and the average length of sentences and words in a document.

Improving the grammar on a sentence level.

Splitting text into paragraphs requires more custom logic. A paragraph might contain a more comprehensive meaning than a sentence.

Finding the words that together form a sentence, or from another viewpoint, detecting sentence boundaries.

Removing sensitive information before a document is shared with others. Deidentification and obfuscation of persons and organizations relies on Named Entity Recognition.

Finding all expressions that refer to the same entity in a text. You can compare this to Named Entity Linking, but it doesn’t necessarily use a knowledge base.

Assigning a unique identity from a knowledge base to a named entity.

Finding strings that contains an indication of time and then extracting a normalized time format out of it.

Parsing text into an address and converting an addresses into geographic coordinates like latitude and longitude.

Extracting price and currency from raw text and normalize it into a standard format.

Abbreviations are an efficient way of writing, but it lowers text comprehension. To solve this, identify the long-form to enrich the short-form.

Identifying named entities is the task of assigning a NER category, like Persons, Locations or Organizations, to words in a sentence.

Breaking text into verb- or noun-phrases result into semantically correct subphrases of a sentence that are deducted from the dependency structure.

Finite size lookup tables might inspire your to build rulebased searches, especially when semantic info like lemma’s and POS- and Dependency tags can be used in the search pattern.

Detecting N-grams results in common multi-word expression with a high probability of occurrence, like the Bi-gram ‘red wine’.