Jan 29

Ansible and Windows Playbooks

Reading time: 3 – 5 minutes

Firstly let me introduce a Windows service called: “Windows Remote Manager” or “WinRM”. This is the Windows feature that allows remote control of Windows machines and many other remote functionalities. In my case I have a Windows 7 laptop with SP1 and PowerShell v3 installed.

Secondly don’t forget that Ansible is developed using Python then a Python library have to manage the WinRM protocol. I’m talking about “pywinrm“. Using this library it’s easy to create simple scripts like that:

#!/usr/bin/env python

import winrm

s = winrm.Session('10.2.0.42', auth=('the_username', 'the_password'))
r = s.run_cmd('ipconfig', ['/all'])
print r.status_code
print r.std_out
print r.std_err

This is a remote call to the command “ipconfig /all” to see the Windows machine network configuration. The output is something like:

$ ./winrm_ipconfig.py 
0

Windows IP Configuration

   Host Name . . . . . . . . . . . . : mini7w
   Primary Dns Suffix  . . . . . . . : 
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : ymbi.net

Ethernet adapter GigaBit + HUB USB:

   Connection-specific DNS Suffix  . : ymbi.net
   Description . . . . . . . . . . . : ASIX AX88179 USB 3.0 to Gigabit Ethernet Adapter
   Physical Address. . . . . . . . . : 00-23-56-1C-XX-XX
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::47e:c2c:8c25:xxxx%103(Preferred) 
   IPv4 Address. . . . . . . . . . . : 10.2.0.42(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.255.192
   Lease Obtained. . . . . . . . . . : mi�rcoles, 28 de enero de 2015 12:41:41
   Lease Expires . . . . . . . . . . : mi�rcoles, 28 de enero de 2015 19:17:56
   Default Gateway . . . . . . . . . : 10.2.0.1
   DHCP Server . . . . . . . . . . . : 10.2.0.1
   DHCPv6 IAID . . . . . . . . . . . : 2063606614
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-15-F7-BF-36-xx-C5-xx-03-xx-xx
   DNS Servers . . . . . . . . . . . : 10.2.0.27
                                       10.2.0.1
   NetBIOS over Tcpip. . . . . . . . : Enabled
...

Of course, it’s possible to run Powershell scripts like the next one which shows the system memory:

$strComputer = $Host
Clear
$RAM = WmiObject Win32_ComputerSystem
$MB = 1048576

"Installed Memory: " + [int]($RAM.TotalPhysicalMemory /$MB) + " MB"

The Python code to run that script is:

#!/usr/bin/env python

import winrm

ps_script = open('scripts/mem.ps1','r').read()
s = winrm.Session('10.2.0.42', auth=('the_username', 'the_password'))
r = s.run_ps(ps_script)
print r.status_code
print r.std_out
print r.std_err

and the output:

$ ./winrm_mem.py 
0
Installed Memory: 2217 MB

In the end it’s time to talk about how to create an Ansible Playbook to deploy anything in a Windows machine. As always the first thing that we need is a hosts file. In the next example there are several ansible variables needed to run Ansible Windows modules on WinRM, all of them are self-explanatory:

[all]
10.2.0.42

[all:vars]
ansible_ssh_user=the_username
ansible_ssh_pass=the_password
ansible_ssh_port=5985 #winrm (non-ssl) port
ansible_connection=winrm

The first basic example could be a simple playbook that runs the ‘ipconfig’ command and registers the output in an Ansible variable to be showed later like a debug information:

- name: test raw module
  hosts: all
  tasks:
    - name: run ipconfig
      raw: ipconfig
      register: ipconfig
    - debug: var=ipconfig

The command and the output to run latest example:

$ ansible-playbook -i hosts ipconfig.yml 

PLAY [test raw module] ******************************************************** 

GATHERING FACTS *************************************************************** 
ok: [10.2.0.42]

TASK: [run ipconfig] ********************************************************** 
ok: [10.2.0.42]

TASK: [debug var=ipconfig] **************************************************** 
ok: [10.2.0.42] => {
    "ipconfig": {
        "invocation": {
            "module_args": "ipconfig", 
            "module_name": "raw"
        }, 
        "rc": 0, 
        "stderr": "", 
        "stdout": "\r\nWindows IP Configuration\r\n\r\n\r\nEthernet adapter GigaBit 
...
        ]
    }
}

PLAY RECAP ******************************************************************** 
10.2.0.42                  : ok=3    changed=0    unreachable=0    failed=0 

As always Ansible have several modules, not only the ‘raw’ module. I committed two examples in my Github account using a module to download URLs and another one that runs Powershell scripts.

My examples are done using Ansible 1.8.2 installed in a Fedora 20. But main problems I’ve found are configuring Windows 7 to accept WinRM connections. Next I attach some references that helped me a lot:

If you want to use my tests code you can connect to my Github: Basic Ansible playbooks for Windows.

Dec 22

Internet of Things Forum 2014 – Plataformas tras el ecosistema

Reading time: < 1 minute El pasado 23 de Noviembre participé en una mesa redonda sobre plataformas del IoT en el marco del IoT forum 2014. Esta jornada esta organizada por BDigital y me dieron la oportunidad de expresar mi opinión en una mesa redonda titulada “Plataformas tras el ecosistema”.

A continuación adjunto el video de la charla:

Si quereis ver el resto de videos de las ponenecias los teneis aquí: Videos de las ponencias del IoT forum 2014.

Dec 01

Short MIIMETIQ definition

Reading time: 2 – 3 minutes

M2MCF and MIIMETIQ

Last months in M2M Cloud Factory we have been working on MIIMETIQ. Last weeks I’ve been thinking about how to define MIIMETIQ shortly and this is my definition, please tell if you can understand something. Of course, you have to know we’re focused in Internet of Things and M2M market.

  • MIIMETIQ is an IoT/M2M framework, so this is the first step to setup to develop your vertical solution.
  • Framework: With a well defined architecture a framework is a set of functions ready to create any application. Everything else is open and adaptable.
  • MIIMETIQ architecture is service oriented and it uses AMQP as a message broker to connect the services.
  • MIIMETIQ has several modules, we define a module as a set of services. Basicly MIIMETIQ have 5 modules:
    • Identity Manager: manage users, groups, roles and all kind of entities the project needs and its security.
    • Assets Manager: a data model manager, the integrator creates the business logics and data models here.
    • Distribution System: this is a set of agnostitc connectivity layers to different types of devices.
    • A E N M: several time series and other signals flow through the AMQP, this data are events and using rules those events could be converted in alarms and some alarms have to be notified to proper services, systems or people.
    • Control Panel UI: this is an administration dashboard, in form of a UI to setup and monitor the most common uses of MIIMETIQ.
  • Using those modules usually the integrators create their own user interface to satisfy customer requiremests. In M2MCF we create those UI using ADUX (Advanced Development User Experience).
  • After configuring MIIMETIQ the integrator has 2 customized APIs to connect their code with MIIMETIQ. One of them is an API REST and another one is AMQP.
  • Finally everything inside MIIMETIQ could be customized, because the flexibility is very important when you have an horizontal solution.
Jul 08

sslsnoop – hacking OpenSSH

Reading time: < 1 minute Using sslsnoop you can dump SSH keys used in a session and decode ciphered traffic. Supported algorithms are: aes128-ctr, aes192-ctr, aes256-ctr, blowfish-cbc, cast128-cbc. Basic sslsnoop information:

 $ sudo sslsnoop # try ssh, sshd and ssh-agent… for various things
 $ sudo sslsnoop-openssh live `pgrep ssh` # dumps SSH decrypted traffic in outputs/
 $ sudo sslsnoop-openssh offline –help # dumps SSH decrypted traffic in outputs/ from a pcap file
 $ sudo sslsnoop-openssl `pgrep ssh-agent` # dumps RSA and DSA keys

Take a look into the project in sslsnoop github page.

Jan 29

Routerboard CRS125-24G-1S-2HnD-IN (Mikrotik) Cloud Switch

Reading time: 1 – 2 minutes

I bought this product a few weeks ago and finally I can enjoy it at home. With this product you have a firewall, gateway, switch and wireless box with:

  • 25x Gigabit Ethernet ports
  • 1x Fiber channel
  • 3G, 4G or any optional USB modem
  • With RouterOS inside you can manage: gateway, firewall, VPN and ad-hoc switching and routing configurations
  • 1000mW high power 2.4GHz 11n wireless AP
CRS125-24G-1S-2HnD-IN

CRS125-24G-1S-2HnD-IN

The official product page is here where you can find brochure in PDF and other useful information.

If you are looking for a powerful product for your SOHO network this is the solution as I like to say ‘this is one of the best communications servers’. It will be very difficult to find some feature or functionality that you can not get from this product. The product is robust and stable with the flexibility of RouterOS.

Jan 24

Conferència: La revolució dels mini-PC: Raspberry PI, Arduino i més

Reading time: 1 – 2 minutes

Ahir al vespre vaig fer una conferència a la FIB (Facultat d’Informàtica de Barcelona) dins de la UPC (Universitat Politècnica de Catalunya). En aquesta xerra vaig estar explicant què és i en que es diferència Arduino i Raspberry PI. A més de presentar tot un conjunt de solucions alternatives i experiències en el tema.

En aquest enllaç podeu trobar les transparències de:  La revolució dels mini-PC: Raspberry PI, Arduino i més. i el video el teniu disponible al servidor de la FIB.

Ara també teniu disponible el video a youtube:

i podeu veure les transparències des d’aquest mateix post:

Espero els vostres feedbacks als comentaris, desitjo que ús sigui útil.

Sep 25

Server send push notifications to client browser without polling

Reading time: 5 – 8 minutes

Nowadays last version of browsers support websockets and it’s a good a idea to use them to connect to server a permanent channel and receive push notifications from server. In this case I’m going to use Mosquitto (MQTT) server behind lighttpd with mod_websocket as notifications server. Mosquitto is a lightweight MQTT server programmed in C and very easy to set up. The best advantage to use MQTT is the possibility to create publish/subscriber queues and it’s very useful when you want to have more than one notification channel. As is usual in pub/sub services we can subscribe the client to a well-defined topic or we can use a pattern to subscribe to more than one topic. If you’re not familiarized with MQTT now it’s the best moment to read a little bit about because that interesting protocol. It’s not the purpose of this post to explain MQTT basics.

A few weeks ago I set up the next architecture just for testing that idea:

mqtt_schema

weboscket gateway to mosquitto mqtt server with javascrit mqtt client

The browser

Now it’s time to explain this proof of concept. HTML page will contain a simple Javascript code which calls mqttws31.js library from Paho. This Javascript code will connect to the server using secure websockets. It doesn’t have any other security measure for a while may be in next posts I’ll explain some interesting ideas to authenticate the websocket. At the end of the post you can download all source code and configuration files. But now it’s time to understand the most important parts of the client code.

client = new Messaging.Client("ns.example.tld", 443, "unique_client_id");
client.onConnectionLost = onConnectionLost;
client.onMessageArrived = onMessageArrived;
client.connect({onSuccess:onConnect, onFailure:onFailure, useSSL:true});

Last part is very simple, the client connects to the server and links some callbacks to defined functions. Pay attention to ‘useSSL’ connect option is used to force SSL connection with the server.

There are two specially interesting functions linked to callbacks, the first one is:

function onConnect() {
  client.subscribe("/news/+/sport", {qos:1,onSuccess:onSubscribe,onFailure:onSubscribeFailure});
}

As you can imagine this callback will be called when the connections is established, when it happens the client subscribes to all channels called ‘/news/+/sports’, for example, ‘/news/europe/sports/’ or ‘/news/usa/sports/’, etc. We can also use, something like ‘/news/#’ and it will say we want to subscribe to all channels which starts with ‘/news/’. If only want to subscribe to one channel put the full name of the channel on that parameter. Next parameter are dictionary with quality of service which is going to use and links two more callbacks.

The second interesting function to understand is:

function onMessageArrived(message) {
  console.log("onMessageArrived:"+message.payloadString);
};

It’s called when new message is received from the server and in this example, the message is printed in console with log method.

The server

I used an Ubuntu 12.04 server with next extra repositories:

# lighttpd + mod_webserver
deb http://ppa.launchpad.net/roger.light/ppa/ubuntu precise main
deb-src http://ppa.launchpad.net/roger.light/ppa/ubuntu precise main

# mosquitto
deb http://ppa.launchpad.net/mosquitto-dev/mosquitto-ppa/ubuntu precise main
deb-src http://ppa.launchpad.net/mosquitto-dev/mosquitto-ppa/ubuntu precise main

With these new repositories you can install required packages:

apt-get install lighttpd lighttpd-mod-websocket mosquitto mosquitto-clients

After installation it’s very easy to run mosquitto in test mode, use a console for that and write the command: mosquitto, we have to see something like this:

# mosquitto
1379873664: mosquitto version 1.2.1 (build date 2013-09-19 22:18:02+0000) starting
1379873664: Using default config.
1379873664: Opening ipv4 listen socket on port 1883.
1379873664: Opening ipv6 listen socket on port 1883.

The configuration file for lighttpd in testing is:

server.modules = (
        "mod_websocket",
)

websocket.server = (
        "/mqtt" => ( 
                "host" => "127.0.0.1",
                "port" => "1883",
                "type" => "bin",
                "subproto" => "mqttv3.1"
        ),
)

server.document-root        = "/var/www"
server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
server.errorlog             = "/var/log/lighttpd/error.log"
server.pid-file             = "/var/run/lighttpd.pid"
server.username             = "www-data"
server.groupname            = "www-data"
server.port                 = 80

$SERVER["socket"] == ":443" {
    ssl.engine = "enable" 
    ssl.pemfile = "/etc/lighttpd/certs/sample-certificate.pem" 
    server.name = "ns.example.tld"
}

Remember to change ‘ssl.pemfile’ for your real certificate file and ‘server.name’ for your real server name. Then restart the lighttpd and validate SSL configuration using something like:

openssl s_client -host ns.example.tld -port 443

You have to see SSL negotiation and then you can try to send HTTP commands, for example: “GET / HTTP/1.0” or something like this. Now the server is ready.

The Test

Now you have to load the HTML test page in your browser and validate how the connections is getting the server and then how the mosquitto console says how it receives the connection. Of course, you can modify the Javascript code to print more log information and follow how the client is connected to MQTT server and how it is subscribed to the topic pattern.

If you want to publish something in MQTT server we could use the CLI, with a command mosquitto_pub:

mosquitto_pub -h ns.example.tld -t '/news/europe/sport' -m 'this is the message about european sports'

Take a look in your browser Javascript consle you have to see how the client prints the message on it. If it fails, review the steps and debug each one to solve the problem. If you need help leave me a message. Of course, you can use many different ways to publish messages, for example, you could use python code to publish messages in MQTT server. In the same way you could subscribe not only browsers to topics, for example, you could subscribe a python code:

import mosquitto

def on_connect(mosq, obj, rc):
    print("rc: "+str(rc))

def on_message(mosq, obj, msg):
    print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

def on_publish(mosq, obj, mid):
    print("mid: "+str(mid))

def on_subscribe(mosq, obj, mid, granted_qos):
    print("Subscribed: "+str(mid)+" "+str(granted_qos))

def on_log(mosq, obj, level, string):
    print(string)

mqttc = mosquitto.Mosquitto("the_client_id")
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_subscribe = on_subscribe

mqttc.connect("ns.example.tld", 1883, 60)
mqttc.subscribe("/news/+/sport", 0)

rc = 0
while rc == 0:
    rc = mqttc.loop()

Pay attention to server port, it isn’t the ‘https’ port (443/tcp) because now the code is using a real MQTT client. The websocket gateway isn’t needed.

The files

  • mqtt.tar.gz – inside this tar.gz you can find all referenced files
Sep 16

RTMP source to HLS (HTTP Live Streaming) Apple

Reading time: 2 – 3 minutes

I just solved a very specific problem and I have to write some notes here to remember the solution. Given a RTMP source we have to stream the content to Apple devices like iPad, iPhone and iPod because RTMP couldn’t be played using Safari browser.

If we need to play streaming on Apple devices the best solution is convert it to HLS and publish generated files using HTTP server.

To solve this issue I use rtmpdump and vlc. Firstly rtmpdump gets video stream from source. Secondly the stream is sent to vlc and finally vlc transcodes de video and audio and outputs small .ts files and one .m3u8 index file.

The command is something like this:

rtmpdump -v -r "$RTMP" | sudo -u xymon vlc -I dummy fd://0 vlc://quit --sout="#transcode{width=320,height=240,fps=25,vcodec=h264,vb=256,venc=x264{aud,profile=baseline,level=30,keyint=30,ref=1,nocabac},acodec=mp3,ab=96,audio-sync,deinterlace,channels=2,samplerate=44100}:std{access=livehttp{seglen=10,delsegs=true,numsegs=5,index=$M3U8,index-url=$TS_URL},mux=ts{use-key-frames},dst=$TSF}"

Variables descriptions are:

RTMP=rtmp://example.tld/path/stream_id
WD=/local_path
TS=live-####.ts
TSF=$WD/$TS
TS_URL=http://example.tld/path/$TS
M3U8=$WD/live.m3u8

Then create an HTML file, for example live.html, with a reference to .m3u8 file, the relevant code of the HTML file is like this:

<video width="320" height="240"><source src="http://example.tld/path/live.m3u8" /></video>

A simple code to public files via HTTP:

python -c "import SimpleHTTPServer;SimpleHTTPServer.test()"

Then we only need to open Safary browser in Apple device and set the proper URL, in our case:

http://example.tld/path/live.html

IMPORTANT NOTE: the audio output have to be with two channels and a sample rate of 44KHz in other cases the audio fails.

Feb 25

Setup a VPN with PPP and SSH

Reading time: < 1 minute Fast trick for linux users, do you know how to setup a VPN using PPP and SSH? of course you can setup a secure tunnel using '-w- or '-W' ssh parameters in last versions of SSH. But in this case I want to share with you this idea:

pppd updetach pty “ssh root@REMOTE_PUB_IP pppd notty 192.168.254.1:192.168.254.2”

I hope it’s useful for you.

Oct 11

Some recommendations about RESTful API design

Reading time: 4 – 6 minutes

I want to recommend to you to watch the YouTube video called RESTful API design of Brian Mulloy. In this post I make an small abstract of the most important ideas of the video, of course from my point of view:

  • Use concrete plural nouns when you are defining resources.
  • Resource URL has to be focused in access collection of elements and specific element. Example:
    • /clients – get all clients
    • /clients/23 – get the client with ID 23
  • Map HTTP methods to maintein elements (CRUD):
    • POST – CREATE
    • GET – READ
    • PUT – UPDATE
    • DELETE – DELETE
  • Workaround, if your REST client doesn’t support HTTP methods, use a parameter called ‘method’ could be a good idea. For example, when you have to use a method HTTP PUT it could be changed by method HTTP GET and the parameter ‘method=put’ in the URL.
  • Sweep complexity behind the ‘?’. Use URL parameters to filter or put some optional information to your request.
  • How to manage errors:
    • Use HTTP response codes to refer error codes. You can find a list of HTTP response codes  in Wikipedia.
    • JSON response example can be like this:
      { 'message':'problem description', 'more_info':'http://api.domain.tld/errors/12345' }
    • Workaround, if REST client doesn’t know how to capture HTTP error codes and raise up an error losing the control of the client, you can use HTTP response code 200 and put ‘response_code’ field in JSON response object. It’s a good idea use this feature as optional across URL parameter ‘supress_response_code=true’.
  • Versioning the API. Use a literal ‘v’ followed by an integer number before the resource reference in the URL. It could be the most simple and powerful solution in this case. Example: /v1/clients/
  • The selection of what information will be returned in the response can be defined in the URL parameters, like in this example: /clients/23?fields=name,address,city
  • Pagination of the response. Use the parameters ‘limit’ and ‘offset’, keep simple. Example: ?limit=10&offset=0
  • Format of the answer, in this case I’m not completely agree with Brian. I prefer to use HTTP header ‘Accept’ than his proposal. Anyway both ideas are:
    • Use HTTP header ‘Accept’ with proper format request in the answer, for example, ‘Accept: application/json’ when you want a JSON response.
    • or, use extension ‘.json’ in URL request to get the response in JSON format.
  • Use Javascript format for date and time information, when you are formatting JSON objects.
  • Sometimes APIs need to share actions. Then we can’t define an action with a noun, in this case use verb. Is common to need actions like: convert, translate, calculate, etc.
  • Searching, there are two cases:
    • Search inside a resource, in this case use parameters to apply filters.
    • Search across multiple resource, here is useful to create the resource ‘search’.
  • Count elements inside a resource, simply add ‘/count’ after the resource. Example: /clients/count
  • As far as you can use a single base URL for all API resources, something like this: ‘http://api.domain.tld’.
  • Authentication, simply use OAuth 2.0
  • To keep your API KISS usually it’s a good idea develop SDK in several languages, where you can put more high level features than in API.
  • Inside an application each resource has its own API but it’s not a good idea publish it to the world, maybe use a virtual API in a layer above it’s more secure and powerful.