Jun 30

Enabling linux kernel to open LOTS of concurrent connections

Reading time: < 1 minute Just a small recipe about how to enable linux kernel to open tons of concurrent connections. Really simple and useful post entry.

echo “10152 65535″ > /proc/sys/net/ipv4/ip_local_port_range
sysctl -w fs.file-max=128000
sysctl -w net.ipv4.tcp_keepalive_time=300
sysctl -w net.core.somaxconn=250000
sysctl -w net.ipv4.tcp_max_syn_backlog=2500
sysctl -w net.core.netdev_max_backlog=2500
ulimit -n 10240

Jan 24

Conferència: La revolució dels mini-PC: Raspberry PI, Arduino i més

Reading time: 1 – 2 minutes

Ahir al vespre vaig fer una conferència a la FIB (Facultat d’Informàtica de Barcelona) dins de la UPC (Universitat Politècnica de Catalunya). En aquesta xerra vaig estar explicant què és i en que es diferència Arduino i Raspberry PI. A més de presentar tot un conjunt de solucions alternatives i experiències en el tema.

En aquest enllaç podeu trobar les transparències de:  La revolució dels mini-PC: Raspberry PI, Arduino i més. i el video el teniu disponible al servidor de la FIB.

Ara també teniu disponible el video a youtube:

i podeu veure les transparències des d’aquest mateix post:

Espero els vostres feedbacks als comentaris, desitjo que ús sigui útil.

Sep 25

Server send push notifications to client browser without polling

Reading time: 5 – 8 minutes

Nowadays last version of browsers support websockets and it’s a good a idea to use them to connect to server a permanent channel and receive push notifications from server. In this case I’m going to use Mosquitto (MQTT) server behind lighttpd with mod_websocket as notifications server. Mosquitto is a lightweight MQTT server programmed in C and very easy to set up. The best advantage to use MQTT is the possibility to create publish/subscriber queues and it’s very useful when you want to have more than one notification channel. As is usual in pub/sub services we can subscribe the client to a well-defined topic or we can use a pattern to subscribe to more than one topic. If you’re not familiarized with MQTT now it’s the best moment to read a little bit about because that interesting protocol. It’s not the purpose of this post to explain MQTT basics.

A few weeks ago I set up the next architecture just for testing that idea:

mqtt_schema

weboscket gateway to mosquitto mqtt server with javascrit mqtt client

The browser

Now it’s time to explain this proof of concept. HTML page will contain a simple Javascript code which calls mqttws31.js library from Paho. This Javascript code will connect to the server using secure websockets. It doesn’t have any other security measure for a while may be in next posts I’ll explain some interesting ideas to authenticate the websocket. At the end of the post you can download all source code and configuration files. But now it’s time to understand the most important parts of the client code.

client = new Messaging.Client("ns.example.tld", 443, "unique_client_id");
client.onConnectionLost = onConnectionLost;
client.onMessageArrived = onMessageArrived;
client.connect({onSuccess:onConnect, onFailure:onFailure, useSSL:true});

Last part is very simple, the client connects to the server and links some callbacks to defined functions. Pay attention to ‘useSSL’ connect option is used to force SSL connection with the server.

There are two specially interesting functions linked to callbacks, the first one is:

function onConnect() {
  client.subscribe("/news/+/sport", {qos:1,onSuccess:onSubscribe,onFailure:onSubscribeFailure});
}

As you can imagine this callback will be called when the connections is established, when it happens the client subscribes to all channels called ‘/news/+/sports’, for example, ‘/news/europe/sports/’ or ‘/news/usa/sports/’, etc. We can also use, something like ‘/news/#’ and it will say we want to subscribe to all channels which starts with ‘/news/’. If only want to subscribe to one channel put the full name of the channel on that parameter. Next parameter are dictionary with quality of service which is going to use and links two more callbacks.

The second interesting function to understand is:

function onMessageArrived(message) {
  console.log("onMessageArrived:"+message.payloadString);
};

It’s called when new message is received from the server and in this example, the message is printed in console with log method.

The server

I used an Ubuntu 12.04 server with next extra repositories:

# lighttpd + mod_webserver
deb http://ppa.launchpad.net/roger.light/ppa/ubuntu precise main
deb-src http://ppa.launchpad.net/roger.light/ppa/ubuntu precise main

# mosquitto
deb http://ppa.launchpad.net/mosquitto-dev/mosquitto-ppa/ubuntu precise main
deb-src http://ppa.launchpad.net/mosquitto-dev/mosquitto-ppa/ubuntu precise main

With these new repositories you can install required packages:

apt-get install lighttpd lighttpd-mod-websocket mosquitto mosquitto-clients

After installation it’s very easy to run mosquitto in test mode, use a console for that and write the command: mosquitto, we have to see something like this:

# mosquitto
1379873664: mosquitto version 1.2.1 (build date 2013-09-19 22:18:02+0000) starting
1379873664: Using default config.
1379873664: Opening ipv4 listen socket on port 1883.
1379873664: Opening ipv6 listen socket on port 1883.

The configuration file for lighttpd in testing is:

server.modules = (
        "mod_websocket",
)

websocket.server = (
        "/mqtt" => ( 
                "host" => "127.0.0.1",
                "port" => "1883",
                "type" => "bin",
                "subproto" => "mqttv3.1"
        ),
)

server.document-root        = "/var/www"
server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
server.errorlog             = "/var/log/lighttpd/error.log"
server.pid-file             = "/var/run/lighttpd.pid"
server.username             = "www-data"
server.groupname            = "www-data"
server.port                 = 80

$SERVER["socket"] == ":443" {
    ssl.engine = "enable" 
    ssl.pemfile = "/etc/lighttpd/certs/sample-certificate.pem" 
    server.name = "ns.example.tld"
}

Remember to change ‘ssl.pemfile’ for your real certificate file and ‘server.name’ for your real server name. Then restart the lighttpd and validate SSL configuration using something like:

openssl s_client -host ns.example.tld -port 443

You have to see SSL negotiation and then you can try to send HTTP commands, for example: “GET / HTTP/1.0” or something like this. Now the server is ready.

The Test

Now you have to load the HTML test page in your browser and validate how the connections is getting the server and then how the mosquitto console says how it receives the connection. Of course, you can modify the Javascript code to print more log information and follow how the client is connected to MQTT server and how it is subscribed to the topic pattern.

If you want to publish something in MQTT server we could use the CLI, with a command mosquitto_pub:

mosquitto_pub -h ns.example.tld -t '/news/europe/sport' -m 'this is the message about european sports'

Take a look in your browser Javascript consle you have to see how the client prints the message on it. If it fails, review the steps and debug each one to solve the problem. If you need help leave me a message. Of course, you can use many different ways to publish messages, for example, you could use python code to publish messages in MQTT server. In the same way you could subscribe not only browsers to topics, for example, you could subscribe a python code:

import mosquitto

def on_connect(mosq, obj, rc):
    print("rc: "+str(rc))

def on_message(mosq, obj, msg):
    print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

def on_publish(mosq, obj, mid):
    print("mid: "+str(mid))

def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed: "+str(mid)+" "+str(granted_qos))

def on_log(mosq, obj, level, string):
    print(string)

mqttc = mosquitto.Mosquitto("the_client_id")
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_subscribe = on_subscribe

mqttc.connect("ns.example.tld", 1883, 60)
mqttc.subscribe("/news/+/sport", 0)

rc = 0
while rc == 0:
    rc = mqttc.loop()

Pay attention to server port, it isn’t the ‘https’ port (443/tcp) because now the code is using a real MQTT client. The websocket gateway isn’t needed.

The files

  • mqtt.tar.gz – inside this tar.gz you can find all referenced files

Sep 20

How to get MP3 file from a WebM video

Reading time: < 1 minute Another title for this post could be: "Getting audio from video clips". Because you could do it with MP4 (Mpeg4), WebM, Mov, FLV, etc. We are going to use ffmpeg to that:

ffmpeg -i input_file.webm -ab 128k -ar 44100 out_file.mp3

The meaning of the parameters:

  • ab: the audio bitrate in bps
  • ar: the audio sample rate in hz

And if you have a directory with a lot of files to convert you could use:

find . -name "*.webm" -print0 |while read -d $'\0' file; do ffmpeg -i "$file" -ab 128k -ar 44100 -y "${file%.webm}.mp3";done

Pay attention to “find” and “while read” commands combinations because we want to support files with spaces.

I hope this is as useful for you as for me.

Sep 16

RTMP source to HLS (HTTP Live Streaming) Apple

Reading time: 2 – 3 minutes

I just solved a very specific problem and I have to write some notes here to remember the solution. Given a RTMP source we have to stream the content to Apple devices like iPad, iPhone and iPod because RTMP couldn’t be played using Safari browser.

If we need to play streaming on Apple devices the best solution is convert it to HLS and publish generated files using HTTP server.

To solve this issue I use rtmpdump and vlc. Firstly rtmpdump gets video stream from source. Secondly the stream is sent to vlc and finally vlc transcodes de video and audio and outputs small .ts files and one .m3u8 index file.

The command is something like this:

rtmpdump -v -r "$RTMP" | sudo -u xymon vlc -I dummy fd://0 vlc://quit --sout="#transcode{width=320,height=240,fps=25,vcodec=h264,vb=256,venc=x264{aud,profile=baseline,level=30,keyint=30,ref=1,nocabac},acodec=mp3,ab=96,audio-sync,deinterlace,channels=2,samplerate=44100}:std{access=livehttp{seglen=10,delsegs=true,numsegs=5,index=$M3U8,index-url=$TS_URL},mux=ts{use-key-frames},dst=$TSF}"

Variables descriptions are:

RTMP=rtmp://example.tld/path/stream_id
WD=/local_path
TS=live-####.ts
TSF=$WD/$TS
TS_URL=http://example.tld/path/$TS
M3U8=$WD/live.m3u8

Then create an HTML file, for example live.html, with a reference to .m3u8 file, the relevant code of the HTML file is like this:

<video width="320" height="240"><source src="http://example.tld/path/live.m3u8" /></video>

A simple code to public files via HTTP:

python -c "import SimpleHTTPServer;SimpleHTTPServer.test()"

Then we only need to open Safary browser in Apple device and set the proper URL, in our case:

http://example.tld/path/live.html

IMPORTANT NOTE: the audio output have to be with two channels and a sample rate of 44KHz in other cases the audio fails.

May 02

Send email notifications from supervisord

Reading time: 1 – 2 minutes

There is a package called superlance which listens supervisord events. If you install it with:

pip install superlance

Then it’s very easy to setup supervisord to send emails when a daemon changes the state because of a crash or something else.

Lines to add to supervisord configuration file:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -a -m email1@example.com
events=PROCESS_STATE

if you want to send notifications only for some applications:

[eventlistener:crashmail]
command=/usr/local/bin/crashmail -p program1 -p group1:program2 -m email1@example.com
events=PROCESS_STATE

Of course, superlance can listen many different event signals from supervisor and can take different actions like call to HTTP URL or send SMS. I want to recommend you to take look to the package documentation it could be useful to understand all the superlance power.

May 01

Relay mail from your server without MTA

Reading time: < 1 minute Sometime you need to send notifications or simply you need to use sendmail command from your server, but you don't want to use a local mail server. Maybe use simple SMTP (ssmtp) could be a good idea to solve this kind of situations. I use to configure SSMTP with a GMail account to send notifications from server different daemons, for example, crontab, supervisord, etc. This is a cookbook configuration for SSMTP and GMail: /etc/ssmtp/ssmtp.conf
root=user@gmail.com
mailhub=smtp.gmail.com:587
rewriteDomain=
hostname=user@gmail.com
UseSTARTTLS=YES
AuthUser=user@gmail.com
AuthPass=password
FromLineOverride=YES

/etc/ssmtp/revaliases

root:username@gmail.com:smtp.gmail.com:587
localusername:username@gmail.com:smtp.gmail.com:587

Installation in ubuntu server is as easy as: apt-get install ssmtp

Feb 25

Setup a VPN with PPP and SSH

Reading time: < 1 minute Fast trick for linux users, do you know how to setup a VPN using PPP and SSH? of course you can setup a secure tunnel using '-w- or '-W' ssh parameters in last versions of SSH. But in this case I want to share with you this idea:

pppd updetach pty “ssh root@REMOTE_PUB_IP pppd notty 192.168.254.1:192.168.254.2”

I hope it’s useful for you.

Dec 21

timegrep: Very useful grep tool

Reading time: < 1 minute Specially when you have to look up inside Postfix logs or Syslog in general it could be the swiss knife tool you need: timegrep.

You have to know that the tool is developed using Python. And is very easy to upgrade or fork the code.