🚀 Building a High-Performance Python Syslog Server with Daily Log Rotation and Emergency Telegram Alerts

In this post, we’ll walk through building a high-performance, multi-threaded syslog server in Python that:

• Listens on multiple UDP ports (like 514 and 1514)

• Stores logs in daily-rotated SQLite databases

• Detects critical syslog messages and sends real-time Telegram alerts

• Handles tens of thousands of messages in queue without data loss

This system is ideal for network engineers and sysadmins managing OLTs, routers, or any syslog-capable devices.

---

🔧 What It Does

• Accepts syslog messages over UDP

• Stores them in a SQLite DB (separate file per day)

• Creates per-device tables (based on source IP)

• Triggers alerts for keywords like pon deregister, reset, rogue, etc.

• Sends alerts to a Telegram group/bot

• Uses Python’s threading and queue for concurrency and performance

---

🧠 Tech Stack

• Python 3

• socket for UDP communication

• sqlite3 for local, fast storage

• requests for Telegram alerts

• threading and queue for parallel log handling

---

📦 Code Features

1. UDP Listener

Each port (514, 1514) runs in its own thread, decoding incoming messages and forwarding them into log_queue.

2. Queue-Based Log Writing

Log entries are pushed into log_queue (max 50,000). A separate log_writer thread reads from this queue and writes to SQLite in batches of 100 entries to optimize performance.

3. Daily Database Rotation

The log writer monitors the date and creates a new DB file per day:

When the date changes, the current DB connection is closed and a new one is opened automatically — no scheduler required.

4. Per-IP Table Creation

Logs are grouped into tables based on the device IP and date:

This avoids a massive flat table and speeds up filtering by device.

5. Real-time Telegram Alerts

When critical patterns are detected in the message content, an alert is pushed to alert_queue, which a separate thread sends to Telegram using:

---

🛡️ Keywords Triggering Alerts

These common OLT/network events are monitored:

• "pon deregister"

• "olt power up", "olt power down"

• "reset", "reboot", "rogue"

• "mac flapp", "loopback"

• "device port updown" and others

You can customize or expand this list easily in the code.

---

🧪 Performance Tips

• Use Queue(maxsize=50000) to buffer bursts of traffic

• Process logs in batches to reduce DB writes

• SO_RCVBUF of 16MB prevents dropped UDP packets

• Use check_same_thread=False when accessing SQLite from threads

• Separate alert and log writers prevent blocking

---

📂 Files Created

• syslog_YYYYMMDD.db — one per day

• Inside each DB: tables like logs_192_168_1_1_20250613

SQLite also creates temporary -wal and -shm files for performance, which is expected behavior.

---

🚀 Running the Server

Download

Donwload the server file and just click. It will run the server.

It starts:

• UDP listeners on 514, 1514

• Log writer and alert threads

• Waits for Ctrl+C to shut down gracefully