Sep 06

Celery logs through syslog

Reading time: 2 – 2 minutes

Celery logs are colorized by default, the first big idea is disable color logs. It’s as easy as setting ‘CELERYD_LOG_COLOR’ to ‘False’ in ‘celery.conf’. The code could be something like this:

celery.conf.update('CELERYD_LOG_COLOR' = False)

Secondly we need a function where we set up a new handler and other settings to celery logging system. For example, the code could be:

from __future__ import absolute_import
from logging import BASIC_FORMAT, Formatter
from logging.handlers import SysLogHandler
from celery.log import redirect_stdouts_to_logger

def setup_log(**args):
    # redirect stdout and stderr to logger
    redirect_stdouts_to_logger(args['logger'])
    # logs to local syslog
    hl = SysLogHandler('/dev/log')
    # setting log level
    hl.setLevel(args['loglevel'])
    # setting log format
    formatter = Formatter(BASIC_FORMAT)
    hl.setFormatter(formatter)
    # add new handler to logger
    args['logger'].addHandler(hl)

Pay attention to ‘redirect_stdouts_to_logger’ it’s used to send all outputs like print’s or something else to syslog.

Thirdly we want to use those settings in our celery tasks, then we have to connect ‘setup_log’ code to some celery signals. Those signals are launched when ‘task_logger’ and ‘logger’ are configured. To connect signals:

from celery.signals import after_setup_task_logger, after_setup_logger

after_setup_logger.connect(setup_log)
after_setup_task_logger.connect(setup_log)

Fourthly we have to get the ‘logger’, we can have more than one if we are interested in records with task context or without it. For example:

logger = get_logger('just_a_name_for_internal_use')
logger_with_task_context = get_task_logger('name_of_the_task_to_be_recorded_in_logs')

Finally we only have to use those loggers with common methods DEBUG, INFO, WARN, ERROR and CRITICAL:

@celery.task
def the_task():
    logger.info('this is a message without task context')
    logger_with_task_context.debug('this record will have the prefix "name_of_the_task_to_be_recorded_in_logs" in syslog')
Aug 28

Technitium MAC Address Changer

Reading time: 1 – 2 minutes

I just want to share with you a small and powerful Windows tool I found in my last trip to US. The best feature IMHO is that permits to change the MAC address of your NIC interface without rebooting, safely and fast. It could be useful when you have a limit time to connect to internet in a free Wi-Fi network; after changing your MAC address you should be like a new device. If you have to do something like this, remember to remove the browser cookies.

Other interesting features of this tool is network presets. You can change your NIC settings very fast just changing a preset profile. As you can see in next screenshot it has a simple chart of your real time network traffic. And finally I want to stand out you can see all your network devices configuration very fast.

technitium MAC address changer screenshot

Technitium MAC Address Changer home page.

May 02

Send email notifications from supervisord

Reading time: 1 – 2 minutes

There is a package called superlance which listens supervisord events. If you install it with:

pip install superlance

Then it’s very easy to setup supervisord to send emails when a daemon changes the state because of a crash or something else.

Lines to add to supervisord configuration file:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -a -m email1@example.com
events=PROCESS_STATE

if you want to send notifications only for some applications:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -p program1 -p group1:program2 -m email1@example.com
events=PROCESS_STATE

Of course, superlance can listen many different event signals from supervisor and can take different actions like call to HTTP URL or send SMS. I want to recommend you to take look to the package documentation it could be useful to understand all the superlance power.

May 01

Relay mail from your server without MTA

Reading time: < 1 minute

Sometime you need to send notifications or simply you need to use sendmail command from your server, but you don’t want to use a local mail server. Maybe use simple SMTP (ssmtp) could be a good idea to solve this kind of situations.

I use to configure SSMTP with a GMail account to send notifications from server different daemons, for example, crontab, supervisord, etc.

This is a cookbook configuration for SSMTP and GMail:

/etc/ssmtp/ssmtp.conf
root=user@gmail.com
mailhub=smtp.gmail.com:587
rewriteDomain=
hostname=user@gmail.com
UseSTARTTLS=YES
AuthUser=user@gmail.com
AuthPass=password
FromLineOverride=YES

/etc/ssmtp/revaliases

root:username@gmail.com:smtp.gmail.com:587
localusername:username@gmail.com:smtp.gmail.com:587

Installation in ubuntu server is as easy as: apt-get install ssmtp

Feb 25

Setup a VPN with PPP and SSH

Reading time: < 1 minute

Fast trick for linux users, do you know how to setup a VPN using PPP and SSH? of course you can setup a secure tunnel using ‘-w- or ‘-W’ ssh parameters in last versions of SSH. But in this case I want to share with you this idea:

pppd updetach pty "ssh root@REMOTE_PUB_IP pppd notty 192.168.254.1:192.168.254.2"

I hope it’s useful for you.

Dec 21

timegrep: Very useful grep tool

Reading time: < 1 minute

Specially when you have to look up inside Postfix logs or Syslog in general it could be the swiss knife tool you need: timegrep.

You have to know that the tool is developed using Python. And is very easy to upgrade or fork the code.

Sep 06

Changing Ubuntu CLI language

Reading time: < 1 minute

My mother tongue is Catalan and of course I speak and understand Spanish very well, but when I’m using a Linux CLI it’s impossible to be agile if the interface is not in English. Then when I need to change Ubuntu interface to English I modify the file /etc/default/locale: 

LANG=en_US.UTF-8
LANGUAGE="en_US:en"
Mar 23

Deep inside AMQP

This entry is part 3 of 4 in the series AMQP and RabbitMQ
Reading time: 5 – 8 minutes

In the next lines I’ll describe with more details the properties and features of AMQP elements. It won’t be an exhaustive description but in my opinion more than enough to start playing with AMQP queues.

Channels

When producers and consumers connects to the broker using a TCP socket after authenticating the connection they establish a channel where AMQP commands are sent. The channel is a virtual path inside a TCP connection between this is very useful because there can be multiple channels inside the TCP connection each channels is identified using an unique ID.

An interesting parameter of a channel is confirmation mode if this is set to true when messages delivered to a exchange finally gets their queues the producer receives an acknowledge message with an UID of the message. This kind of messages are asynchronous and permits to a producer send the next message when it is still waiting the ACK message. Of course if the message cannot be stored and it is lost the producer receives a NACK (not acknowledged) message.

Producers

Maybe this is the most simple part of the system. Producers only need to negotiate the authentication across a TCP connection create a channel and then publish all messages that want with its corresponding routing key. Of course, producers can create exchanges, queues and then bind them. But usually this is not a good idea is much more secure do this from consumers. Because when a producers try to send a message to a broker and doesn’t have the needed exchange then message will be lost. Usually consumers are connected all time and subscribed to queues and producers only connect to brokers when they need to send messages.

Consumers

When a consumer connects to a queue usually uses a command called basic.consume to subscribe the channel to a queue, then every time subscribed queue has a new message it is sent to consumer after last message is consumed, or rejected.

If consumer only want to receive one message without a subscription it can use the command basic.get.This is like a poll method. In fact, the consumer only gets a message each time it sends the command.

You can get the best throughput using basic.consume command because is more efficient than poll every time the consumer wants another message.

When more than one consumer was connected to a queue, messages are distributed in a round-robin. After the message is delivered to a consumer this send an acknowledge message and then queue send another message to next consumer. If the consumer sends a reject message the same message is sent to next consumer.

There are two types of acknowledgements:

  • basic.ack: this is the message that sends consumer to queue to acknowledge the reception of a message
  • auto_ack: this is a parameter we can set when consumer subscribes to a queue. The setting assumes ACK message from consumer and then queue sends next message without waiting the ACK message.

The message basic.reject is sent when the consumer wants to reject a received message. This message discards the message and it is lost. If we want to requeue the message we can set the parameter requeue=true when sent a reject message.

When the queue is created there can be a parameter called dead letter set to true, then consumer rejects a message with the parameter requeue=false the message is queued to a new queue called  dead letter. This is very useful because after all we can go tho that queue an inspect the message rejection reason.

Queues

Both consumers and producers can create a queue using queue.declare command. The most natural way is create queues from consumers and then bind it to an exchange. The consumers needs a free channel to create a queue, if a channel is subscribed to a queue, the channel is busy and cannot create new queues. When a queue is created usually we use a name to identify the queue, if the name is not specified it’s randomly generated. This is useful when create temporary and anonymous queues for RPC-over-AMQP.

Parameters we can set when create a new queue:

  • exclusive – this setting makes a queue private and is only accessible from your application. Only one consumer can connect to a queue.
  • auto-delete – when last consumer unsubscribes from queue the queue is removed.
  • passive – when create a queue that exists the server returns successfully or returns fail if parameters don’t match. If passive parameter is set and we create a queue that exists always returns success but if the queue doesn’t exist it is not created.
  • durable – the queue can persist when the services reboots.

Exchange and binding

In the first post of the serie we talked about different exchange types as you can remember these types are: direct, fanout and topic. And the most important parameter to set when producer sends a message is the routing key this is used to route the message to a queue.

Once we have declared an exchange this can be related with a queue using a binding command: queue_bind. The relation between them is made using the routing key or a pattern based in routing key. When exchange has type fanout the routing key or patterns are not needed.

Some pattern examples can be: log.*, message.* and #.

The most important exchange parameters are:

  • type: direct, fanout and topic.
  • durable: makes an exchange persistent to reboots.

Broker and virtual hosts

A broker is a container where exhanges, bindings and queues are created. Usually we can define more than one virtual brokers in the same server. Virtual brokers are also called virtual hosts. The users, permissions and something else related to a Broker cannot be used from another one. This is very useful because we can create multiple brokers in the same physical server like multi-domain web server and when some of this virtual hosts is too big it can be migrated to another physical server and it can be clustered if it is required.

Messages

An AMQP message is a binary without a fixed size and format. Each application can set it’s own messages. The AMQP broker only will add small headers to be routed among different queues as fast as possible.

Messages are not persistent inside a broker unless the producer sets the parameter persistent=true. In the other way the messages needs to be stored in durable exchanges and durable queues to persist in the broker when it is restarted. Of course when the messages are persistent these must be wrote to disk and the throughput will fall down. Then maybe sometimes create persistent messages is not a good idea.