🚀 Building a High-Performance Python Syslog Server with Daily Log Rotation and Emergency Telegram Alerts

In this post, we’ll walk through building a high-performance, multi-threaded syslog server in Python that:

• Listens on multiple UDP ports (like 514 and 1514)

• Stores logs in daily-rotated SQLite databases

• Detects critical syslog messages and sends real-time Telegram alerts

• Handles tens of thousands of messages in queue without data loss

This system is ideal for network engineers and sysadmins managing OLTs, routers, or any syslog-capable devices.

---

🔧 What It Does

• Accepts syslog messages over UDP

• Stores them in a SQLite DB (separate file per day)

• Creates per-device tables (based on source IP)

• Triggers alerts for keywords like pon deregister, reset, rogue, etc.

• Sends alerts to a Telegram group/bot

• Uses Python’s threading and queue for concurrency and performance

---

🧠 Tech Stack

• Python 3

• socket for UDP communication

• sqlite3 for local, fast storage

• requests for Telegram alerts

• threading and queue for parallel log handling

---

📦 Code Features

1. UDP Listener

Each port (514, 1514) runs in its own thread, decoding incoming messages and forwarding them into log_queue.

2. Queue-Based Log Writing

Log entries are pushed into log_queue (max 50,000). A separate log_writer thread reads from this queue and writes to SQLite in batches of 100 entries to optimize performance.

3. Daily Database Rotation

The log writer monitors the date and creates a new DB file per day:

When the date changes, the current DB connection is closed and a new one is opened automatically — no scheduler required.

4. Per-IP Table Creation

Logs are grouped into tables based on the device IP and date:

This avoids a massive flat table and speeds up filtering by device.

5. Real-time Telegram Alerts

When critical patterns are detected in the message content, an alert is pushed to alert_queue, which a separate thread sends to Telegram using:

---

🛡️ Keywords Triggering Alerts

These common OLT/network events are monitored:

• "pon deregister"

• "olt power up", "olt power down"

• "reset", "reboot", "rogue"

• "mac flapp", "loopback"

• "device port updown" and others

You can customize or expand this list easily in the code.

---

🧪 Performance Tips

• Use Queue(maxsize=50000) to buffer bursts of traffic

• Process logs in batches to reduce DB writes

• SO_RCVBUF of 16MB prevents dropped UDP packets

• Use check_same_thread=False when accessing SQLite from threads

• Separate alert and log writers prevent blocking

---

📂 Files Created

• syslog_YYYYMMDD.db — one per day

• Inside each DB: tables like logs_192_168_1_1_20250613

SQLite also creates temporary -wal and -shm files for performance, which is expected behavior.

---

🚀 Running the Server

Download

Donwload the server file and just click. It will run the server.

It starts:

• UDP listeners on 514, 1514

• Log writer and alert threads

• Waits for Ctrl+C to shut down gracefully

Building a Simple Syslog Server with Python and Django: A Comprehensive Guide

In today's world, network monitoring and log management are essential for maintaining and troubleshooting network devices. If you're looking to build a reliable and customizable syslog server, Python and Django provide a powerful and flexible solution. This guide will walk you through creating a simple syslog server that not only captures logs but also supports exporting them to CSV with filters.

Introduction to the Syslog Server

A syslog server is crucial for collecting and managing logs from network devices. Our server is specifically designed to work seamlessly with MikroTik routers but is versatile enough to support other devices. This server captures log entries, processes them, and offers the ability to export filtered logs to CSV.

Key Features

• Log Collection: Capture syslog messages from various network devices.
• Auto Delete older than 30 days of data to faster performance
• Filtering Options: Filter logs by host IP, message content, and specific timestamps.
• Export to CSV: Easily export filtered logs to CSV for further analysis.

Setting Up the Syslog Server

1. Prerequisites

Before we dive into the implementation, make sure you have the following:

• Python (3.x)
• Django
• Pandas, numpy, openpyxl library for CSV export
• Basic understanding of Python and Django

Download the Syslog Server

You can download the complete code for this syslog server from the following link:

Drive link: Download

Instalation: 

Open your mikrotik and configure remote log first. My router ip 192.168.10.252. Run below command in your mikrotik. 

    /system logging action

    set 3 remote=192.168.0.104 src-address=192.168.10.113

    /system logging

    add action=remote topics=info

Download and extract the rar file. then open folder syslog_project. In project folder just run the following command with the server ip (which is your interface ip)

           python manage.py runserver 192.168.0.104:8000

It should run like below.

Now open browser and hit the link: http://192.168.0.104:8000/logs/

Conclusion

This guide provided a step-by-step approach to building a simple syslog server using Python and Django. The server captures logs, supports filtering by IP, message, and timestamp, and allows exporting the filtered data to CSV. This setup is especially useful for managing logs from MikroTik devices but can be adapted for other network devices as well.
Feel free to explore and modify the code to suit your specific requirements. Happy logging!