Redis Log Handler

smarter.lib.logging.redis_log_handler

This module provides a custom logging handler for publishing log records to Redis channels, enabling real-time log streaming for distributed systems. It is designed to work with context-aware job IDs, allowing logs to be associated with specific jobs or tasks (such as Celery tasks), as well as to a global log channel for system-wide log aggregation.

This handler is used for the Terminal Emulator real-time log view in the Smarter web console.

Main Components

smarter.lib.logging.redis_log_handler.RedisLogHandler: A custom logging handler that publishes log records to Redis channels, supporting both job-specific and global log streams.
smarter.lib.logging.redis_log_handler.job_id_factory: Utility function to generate unique job IDs for associating logs with specific jobs or tasks.
smarter.lib.logging.redis_log_handler.user_id_context: Context variable for tracking the current job ID within the logging context.
smarter.lib.logging.redis_log_handler.GLOBAL_LOG_CHANNEL: The Redis channel name used for publishing all logs globally.

Features

Asynchronous log publishing to Redis using a background worker thread and batching for efficiency.
Support for both job-specific and global log channels.
Graceful shutdown and log flushing on process exit.
Handles dropped logs if the internal queue is full, with periodic reporting.

Example Usage

#
# configure the Django logging to use the RedisLogHandler
#
LOGGING = {
    "handlers": {
        "default": {
            "level": smarter_settings.log_level_name,
            "class": "logging.StreamHandler",
            "formatter": "timestamped",
        },
        "redis": {
            "level": smarter_settings.log_level_name,
            "class": "smarter.lib.logging.RedisLogHandler",  # <--- Use the RedisLogHandler
            "formatter": "timestamped",
        },
    }

Attention

The following is technically possible but not a recommended practice. It demonstrates how to manually configure logging to use the RedisLogHandler outside of a Django settings context, such as in a standalone script or a Celery task. In most cases, it’s better to configure logging through the Django settings for consistency and maintainability.

#
# manually configure logging to use RedisLogHandler
# NOTE: this is technically possible but not a great idea.
#
import logging
from smarter.lib.logging.redis_log_handler import RedisLogHandler, user_id_context, job_id_factory

# Set the current job ID (typically in a Celery task or similar context)
user_id_context.set(job_id_factory("task"))

# Configure the logger
logger = logging.getLogger("my_logger")
logger.setLevel(logging.INFO)
logger.addHandler(RedisLogHandler())

logger.info("This log will be published to Redis.")

class smarter.lib.logging.redis_log_handler.RedisLogHandler(level=0)[source]

Bases: Handler

Custom logging handler that publishes log records to Redis channels for real-time log streaming.

This handler supports both job-specific and global log channels:

If a job ID is present in the user_id_context context variable, the log record is published to
the Redis channel logs:{user_id}, where {user_id} is the unique identifier for the job or task.
All log records are also published to the global channel defined by GLOBAL_LOG_CHANNEL (logs:global),
which can be used for system-wide log aggregation or UI log feeds.

The handler is designed to be used in distributed or asynchronous environments (e.g., Celery tasks), where each job or task can have its own log stream. Log records are enqueued and published asynchronously by a background worker thread for efficiency and non-blocking behavior.

If the internal log queue is full, log records may be dropped. The number of dropped logs is tracked by the class variable dropped_logs, and a message is printed every 100 dropped logs.