lang-detect

lang-detect is a super handy Python tool that helps you figure out what language a piece of text is in. Imagine you have a small chunk of Unicode text, and you want to know if it's in English, Spanish, or maybe Japanese—this tool does just that without needing any extra libraries!

Languages Supported

Right now, lang-detect can spot languages like German (de), English (en), Spanish (es), French (fr), Italian (it), Japanese (ja), Dutch (nl), Polish (pl), Russian (ru), Simplified Chinese (zh-hans), Traditional Chinese (zh-hant), and Cantonese (zh-yue). Pretty cool, right?

How It Works

After running some tests, we noticed that the tool works best with longer sentences. So if you're trying it out, keep that in mind! The magic happens in the Basic Multilingual Plane of Unicode encoding, which means we can actually expand the number of languages it supports in the future.

N-Gram Vector Representation

For each language, we use something called a uniformed n-gram vector to represent the language itself. You can check this out in the data folder of the project. When you input a text for detection, lang-detect generates an n-gram vector for your text and compares it to the vectors for each supported language using cosine similarity.

The Corpus Used

You might wonder where we get our data from. Well, we use feature articles from Wikipedia as our corpus! This helps ensure accuracy when identifying languages.

Getting Started

If you're ready to give it a shot, just head over to your project root and run this command:

bin/langdetect YOUR_SENTENCE_HERE

If you want to download lang-detect now!