Center for Tamil Natural Language Processing Research

🌟𝐁𝐮𝐢𝐥𝐝𝐢𝐧𝐠 𝐚 𝐒𝐫𝐢 𝐋𝐚𝐧𝐤𝐚𝐧 𝐓𝐚𝐦𝐢𝐥 𝐍𝐚𝐦𝐞𝐝 𝐄𝐧𝐭𝐢𝐭𝐲 𝐑𝐞𝐜𝐨𝐠𝐧𝐢𝐭𝐢𝐨𝐧 𝐃𝐚𝐭𝐚𝐬𝐞𝐭 𝐟𝐨𝐫 𝐋𝐨𝐰-𝐑𝐞𝐬𝐨𝐮𝐫𝐜𝐞 𝐍𝐋𝐏

The growth of Large Language Models (LLMs) and multilingual NLP systems has significantly improved language technologies across major global languages. However, low-resource languages such as Sri Lankan Tamil still face a severe lack of high-quality annotated datasets—especially for foundational tasks like Named Entity Recognition (NER).

To address this gap, we developed the Srilankan-Tamil-NER Dataset, a Tamil NER dataset designed specifically for Sri Lankan Tamil linguistic and contextual usage.

This dataset is intended to support:

• Tamil NER research
• Indic language fine-tuning
• Information extraction systems
• Retrieval-Augmented Generation (RAG)
• Tamil LLM adaptation
• Domain-specific AI systems for Sri Lanka

𝗪𝗵𝘆 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹 𝗡𝗘𝗥 𝗠𝗮𝘁𝘁𝗲𝗿𝘀

Named Entity Recognition (NER) is a core NLP task that identifies and classifies entities such as:

• Person names
• Locations
• Organizations
• Dates
• Miscellaneous entities

NER acts as a foundational layer for many downstream NLP systems including:

• Question answering
• Search systems
• Chatbots
• Document intelligence
• Machine translation
• Knowledge graph generation

For Tamil — particularly Sri Lankan Tamil — publicly available annotated corpora remain extremely limited. Existing multilingual datasets often underrepresent regional linguistic variations, local named entities, and culturally contextual terminology.

Most existing NER systems for Tamil are trained on datasets originating from Indian Tamil corpora, leaving significant gaps in handling:

• Sri Lankan Tamil vocabulary
• Local organization names
• Sri Lankan place names
• Government and institutional terminology

Our dataset aims to bridge this gap.

𝗔𝗯𝗼𝘂𝘁 𝘁𝗵𝗲 𝗗𝗮𝘁𝗮𝘀𝗲𝘁

𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗡𝗮𝗺𝗲:
Srilankan-Tamil-NER Dataset

The primary goal of this dataset is to create a high-quality manually curated Named Entity Recognition corpus for Sri Lankan Tamil under CTNLPR.

The dataset is structured to support fine-tuning transformer-based multilingual models such as:

• IndicNER
• mBERT
• XLM-RoBERTa
• MuRIL
• IndicBERT

𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗦𝘁𝗮𝘁𝗶𝘀𝘁𝗶𝗰𝘀

• B-PER (Person): 4,533
• B-LOC (Location): 8,110
• B-ORG (Organization): 3,369
• Total Entities: 16,012

𝗗𝗮𝘁𝗮𝘀𝗲𝘁 𝗣𝗿𝗲𝗽𝗮𝗿𝗮𝘁𝗶𝗼𝗻 𝗣𝗶𝗽𝗲𝗹𝗶𝗻𝗲

Creating a Tamil NER dataset involves significantly more than simple annotation.

The preparation workflow included multiple stages:

1. 𝑫𝒂𝒕𝒂 𝑪𝒐𝒍𝒍𝒆𝒄𝒕𝒊𝒐𝒏

The raw Tamil text corpus was collected from the Noolaham corpus and other relevant publicly available Sri Lankan Tamil textual sources.

Special attention was given to:

• Local linguistic relevance
• Entity diversity
• Sentence quality
• Contextual richness

The objective was to capture realistic Sri Lankan Tamil usage patterns rather than synthetic or translated text.

2. 𝑶𝑪𝑹 𝒂𝒏𝒅 𝑻𝒆𝒙𝒕 𝑵𝒐𝒓𝒎𝒂𝒍𝒊𝒛𝒂𝒕𝒊𝒐𝒏

Tamil NLP pipelines often begin with scanned or image-based documents.

As part of our broader Tamil document intelligence workflow, OCR-extracted Tamil text underwent:

• Unicode normalization
• Punctuation cleaning
• Whitespace normalization
• Invalid character filtering
• OCR noise reduction

OCR-related preprocessing becomes extremely important because Tamil script errors can propagate heavily into token classification systems.

3. 𝑵𝒂𝒎𝒆𝒅 𝑬𝒏𝒕𝒊𝒕𝒚 𝑨𝒏𝒏𝒐𝒕𝒂𝒕𝒊𝒐𝒏

The dataset was manually annotated using BIO tagging format.

Entity Types:

• B-PER — Beginning of person entity
• I-PER — Inside person entity
• B-LOC — Beginning of location entity
• I-LOC — Inside location entity
• B-ORG — Beginning of organization entity
• I-ORG — Inside organization entity
• O — Non-entity token

Example:

இராமநாதன் → B-PER
யாழ்ப்பாணம் → B-LOC
பல்கலைக்கழகம் → B-ORG

𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲𝘀 𝗶𝗻 𝗦𝗿𝗶 𝗟𝗮𝗻𝗸𝗮𝗻 𝗧𝗮𝗺𝗶𝗹 𝗡𝗘𝗥

Building a Tamil NER dataset introduced several language-specific challenges.

• Morphological complexity
• OCR noise
• Unicode inconsistencies
• Token boundary detection
• Subword alignment
• Limited benchmark corpora

𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗨𝘀𝗲 𝗖𝗮𝘀𝗲𝘀

This dataset can support:

• Tamil NER
• OCR post-processing
• Semantic search systems
• RAG pipelines
• Tamil chatbots
• Government document AI
• Knowledge graph generation

The Srilankan-Tamil-NER Dataset, developed under CTNLPR, represents an important step toward strengthening the Sri Lankan Tamil NLP ecosystem through high-quality entity annotation and linguistically relevant corpus preparation.

#𝑆𝑟𝑖𝐿𝑎𝑛𝑘𝑎𝑛𝑇𝑎𝑚𝑖𝑙 #𝑇𝑎𝑚𝑖𝑙𝑁𝐸𝑅 #𝑇𝑎𝑚𝑖𝑙𝑁𝐿𝑃 #𝑁𝑎𝑚𝑒𝑑𝐸𝑛𝑡𝑖𝑡𝑦𝑅𝑒𝑐𝑜𝑔𝑛𝑖𝑡𝑖𝑜𝑛 #𝐿𝑜𝑤𝑅𝑒𝑠𝑜𝑢𝑟𝑐𝑒𝑁𝐿𝑃 #𝐼𝑛𝑑𝑖𝑐𝑁𝐿𝑃 #𝑀𝑢𝑅𝐼𝐿 #𝑚𝐵𝐸𝑅𝑇 #𝑋𝐿𝑀𝑅 #𝐵𝐼𝑂𝑡𝑎𝑔𝑔𝑖𝑛𝑔 #𝐿𝐿𝑀 #𝑅𝐴𝐺 #𝑆𝑒𝑚𝑎𝑛𝑡𝑖𝑐𝑆𝑒𝑎𝑟𝑐ℎ #𝐾𝑛𝑜𝑤𝑙𝑒𝑑𝑔𝑒𝐺𝑟𝑎𝑝ℎ𝑠 #𝐴𝐼𝐸𝑛𝑔𝑖𝑛𝑒𝑒𝑟𝑖𝑛𝑔 #𝐷𝑜𝑐𝑢𝑚𝑒𝑛𝑡𝐼𝑛𝑡𝑒𝑙𝑙𝑖𝑔𝑒𝑛𝑐𝑒 #𝐸𝑛𝑡𝑖𝑡𝑦𝐸𝑥𝑡𝑟𝑎𝑐𝑡𝑖𝑜𝑛 #𝑂𝐶𝑅 #𝑇𝑟𝑎𝑛𝑠𝑓𝑜𝑟𝑚𝑒𝑟𝑀𝑜𝑑𝑒𝑙𝑠 #𝐶𝑜𝑚𝑝𝑢𝑡𝑎𝑡𝑖𝑜𝑛𝑎𝑙𝐿𝑖𝑛𝑔𝑢𝑖𝑠𝑡𝑖𝑐𝑠 #𝐶𝑇𝑁𝐿𝑃𝑅