oriolrius.cat

Des del 2000 compartiendo sobre…

Celery logs through syslog

Reading time: 2 – 2 minutes

Celery logs are colorized by default, the first big idea is disable color logs. It’s as easy as setting ‘CELERYD_LOG_COLOR’ to ‘False’ in ‘celery.conf’. The code could be something like this:

celery.conf.update('CELERYD_LOG_COLOR' = False)

Secondly we need a function where we set up a new handler and other settings to celery logging system. For example, the code could be:

from __future__ import absolute_import
from logging import BASIC_FORMAT, Formatter
from logging.handlers import SysLogHandler
from celery.log import redirect_stdouts_to_logger

def setup_log(**args):
    # redirect stdout and stderr to logger
    redirect_stdouts_to_logger(args['logger'])
    # logs to local syslog
    hl = SysLogHandler('/dev/log')
    # setting log level
    hl.setLevel(args['loglevel'])
    # setting log format
    formatter = Formatter(BASIC_FORMAT)
    hl.setFormatter(formatter)
    # add new handler to logger
    args['logger'].addHandler(hl)

Pay attention to ‘redirect_stdouts_to_logger’ it’s used to send all outputs like print’s or something else to syslog.

Thirdly we want to use those settings in our celery tasks, then we have to connect ‘setup_log’ code to some celery signals. Those signals are launched when ‘task_logger’ and ‘logger’ are configured. To connect signals:

from celery.signals import after_setup_task_logger, after_setup_logger

after_setup_logger.connect(setup_log)
after_setup_task_logger.connect(setup_log)

Fourthly we have to get the ‘logger’, we can have more than one if we are interested in records with task context or without it. For example:

logger = get_logger('just_a_name_for_internal_use')
logger_with_task_context = get_task_logger('name_of_the_task_to_be_recorded_in_logs')

Finally we only have to use those loggers with common methods DEBUG, INFO, WARN, ERROR and CRITICAL:

@celery.task
def the_task():
    logger.info('this is a message without task context')
    logger_with_task_context.debug('this record will have the prefix "name_of_the_task_to_be_recorded_in_logs" in syslog')

One thought on “Celery logs through syslog

  1. I am an investor of gate io, I have consulted a lot of information, I hope to upgrade my investment strategy with a new model. Your article creation ideas have given me a lot of inspiration, but I still have some doubts. I wonder if you can help me? Thanks.

Últimas entradas

Avui ja n’he fet 20

Reading time: < 1 minute Aprofito aquesta entrada per comentar una nova efemèride s’ha produït avui mateix. Ja que després de 46 anys menys dos dies he visitat de nou l’hospital on vaig néixer. Aquest cop per fer-me unes radiografies

Leer más »

Resumen 2022

Reading time: 26 – 42 minutes El 2022 empezó esquiando en Candanchú. Un fin de año en familia y con una COVID que se iba despidiendo después de habernos cambiado la vida a todos. Hemos tenido la fortuna de que

Leer más »

Deploying gotop with Ansible

Reading time: 1 – 2 minutes Gotop is a terminal based graphical activity monitor inspired by gtop and vtop; it’s available at: https://github.com/xxxserxxx/gotop/ I published a role in Ansible Galaxy for deploying gotop in Linux servers. The role page in

Leer más »
Archivo