File-based Streaming Log Handler

Custom logging handlers for the Smarter application.

This module provides a custom logging handler, StreamingFileHandler, which writes log records to a file in real-time. This is useful for capturing logs for individual jobs or processes, especially when logs need to be streamed or persisted separately from the main application log.

Classes

StreamingFileHandler

A logging handler that writes log records to a temporary file, one per job.

Examples

To use the streaming file handler in your logging configuration:

import logging
from smarter.lib.logging.streaming_file_handler import StreamingFileHandler

handler = StreamingFileHandler(job_id="my-job-123")
logger = logging.getLogger("my_job_logger")
logger.addHandler(handler)
logger.info("This log will be written to a job-specific file.")

The log file will be created in the system temporary directory under a “logs” subdirectory.

class smarter.lib.logging.streaming_file_handler.StreamingFileHandler(job_id)

Bases: Handler

Logging handler that writes log records to a job-specific file in real-time.

This handler creates (or appends to) a log file in the system temporary directory under a “logs” subdirectory. Each handler instance writes to a file named after the provided job ID. Log records are formatted and written immediately as they are emitted.

Parameters:

job_id (str) – Unique identifier for the job or process. Used as the log file name (e.g., <job_id>.log).

path

The full path to the log file where records are written.

Type:

str

Examples

>>> handler = StreamingFileHandler(job_id="my-job-123")
>>> import logging
>>> logger = logging.getLogger("my_job_logger")
>>> logger.addHandler(handler)
>>> logger.info("This log will be written to a job-specific file.")

The log file will be created in the system temporary directory under a “logs” subdirectory.

__init__(job_id)

Initializes the instance - basically setting the formatter to None and the filter list to empty.

_at_fork_reinit()

acquire()

Acquire the I/O thread lock.

addFilter(filter)

Add the specified filter to this handler.

close()

Tidy up any resources used by the handler.

This version removes the handler from an internal map of handlers, _handlers, which is used for handler lookup by name. Subclasses should ensure that this gets called from overridden close() methods.

createLock()

Acquire a thread lock for serializing access to the underlying I/O.

emit(record)

Write a log record to the job-specific log file.

This method is called by the logging framework for each log record. It formats the record using the handler’s formatter and appends it to the log file associated with this handler’s job ID. The log file is opened in append mode and flushed after each write to ensure real-time streaming.

Parameters:

record (logging.LogRecord) – The log record to be written to the file. This object contains all information about the event being logged.

Notes

If the log file or directory does not exist, it will be created automatically. The file is opened in UTF-8 encoding and each log entry is written on a new line.

Examples

This method is typically called by the logging framework and not used directly.

filter(record)

Determine if a record is loggable by consulting all the filters.

The default is to allow the record to be logged; any filter can veto this by returning a false value. If a filter attached to a handler returns a log record instance, then that instance is used in place of the original log record in any further processing of the event by that handler. If a filter returns any other true value, the original log record is used in any further processing of the event by that handler.

If none of the filters return false values, this method returns a log record. If any of the filters return a false value, this method returns a false value.

Changed in version 3.2: Allow filters to be just callables.

Changed in version 3.12: Allow filters to return a LogRecord instead of modifying it in place.

flush()

Ensure all logging output has been flushed.

This version does nothing and is intended to be implemented by subclasses.

format(record)

Format the specified record.

If a formatter is set, use it. Otherwise, use the default formatter for the module.

get_name()

handle(record)

Conditionally emit the specified logging record.

Emission depends on filters which may have been added to the handler. Wrap the actual emission of the record with acquisition/release of the I/O thread lock.

Returns an instance of the log record that was emitted if it passed all filters, otherwise a false value is returned.

handleError(record)

Handle errors which occur during an emit() call.

This method should be called from handlers when an exception is encountered during an emit() call. If raiseExceptions is false, exceptions get silently ignored. This is what is mostly wanted for a logging system - most users will not care about errors in the logging system, they are more interested in application errors. You could, however, replace this with a custom handler if you wish. The record which was being processed is passed in to this method.

property name

release()

Release the I/O thread lock.

removeFilter(filter)

Remove the specified filter from this handler.

setFormatter(fmt)

Set the formatter for this handler.

setLevel(level)

Set the logging level of this handler. level must be an int or a str.

set_name(name)