Jan 30

New Ansible Role uploaded to Ansible Galaxy

Reading time: 2 – 2 minutes

A long time ago I wrote an entry post about how to set up the SMTP in linux boxes using a relay system you can find the post here: Relay mail from your server without MTA. Remember that SSMTP is not a SMTP service for your system but it’s more than enough for all servers that don’t work as a mail servers. Historically Unix/Linux uses sendmail command to send system notifications but usually this mails are lost because system configurations are not completed. My advice in this sense is use SSMTP.

In the past I used to use SSMTP with a GMail account but security constraints in Google mail services make it difficult to configure today. The new alternative is set up a free Mandrill account as a relay host. Mandrill is a Mailchimp service that allows you to send a lot of emails without problems and there is a free account that allows to send up to 12.000 mails per month free, more than enough usually. If you don’t know how to set up a Mailchimp account the best option to learn how to do it is follow the support documentation it’s very good IMHO.

When you have a lot of linux machines to administer you need something fastly replicable. As you know use Ansible is a very good option. Then I developed a new Ansible role to set up Mandrill accounts to SSMTP services massively using Ansible.

The Ansible role has been uploaded here: ssmtp-mandrill and the source code of the roles is in my github. Remember to install the role in your Ansible is easy:

ansible-galaxy install oriol.rius.ssmtp-mandrill

Then you only need to create your own playbook using the role and don’t forget to setup the variables with the Mandrill account settings.

Jan 29

Ansible and Windows Playbooks

Reading time: 3 – 5 minutes

Firstly let me introduce a Windows service called: “Windows Remote Manager” or “WinRM”. This is the Windows feature that allows remote control of Windows machines and many other remote functionalities. In my case I have a Windows 7 laptop with SP1 and PowerShell v3 installed.

Secondly don’t forget that Ansible is developed using Python then a Python library have to manage the WinRM protocol. I’m talking about “pywinrm“. Using this library it’s easy to create simple scripts like that:

#!/usr/bin/env python

import winrm

s = winrm.Session('10.2.0.42', auth=('the_username', 'the_password'))
r = s.run_cmd('ipconfig', ['/all'])
print r.status_code
print r.std_out
print r.std_err

This is a remote call to the command “ipconfig /all” to see the Windows machine network configuration. The output is something like:

$ ./winrm_ipconfig.py 
0

Windows IP Configuration

   Host Name . . . . . . . . . . . . : mini7w
   Primary Dns Suffix  . . . . . . . : 
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : ymbi.net

Ethernet adapter GigaBit + HUB USB:

   Connection-specific DNS Suffix  . : ymbi.net
   Description . . . . . . . . . . . : ASIX AX88179 USB 3.0 to Gigabit Ethernet Adapter
   Physical Address. . . . . . . . . : 00-23-56-1C-XX-XX
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::47e:c2c:8c25:xxxx%103(Preferred) 
   IPv4 Address. . . . . . . . . . . : 10.2.0.42(Preferred) 
   Subnet Mask . . . . . . . . . . . : 255.255.255.192
   Lease Obtained. . . . . . . . . . : mi�rcoles, 28 de enero de 2015 12:41:41
   Lease Expires . . . . . . . . . . : mi�rcoles, 28 de enero de 2015 19:17:56
   Default Gateway . . . . . . . . . : 10.2.0.1
   DHCP Server . . . . . . . . . . . : 10.2.0.1
   DHCPv6 IAID . . . . . . . . . . . : 2063606614
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-15-F7-BF-36-xx-C5-xx-03-xx-xx
   DNS Servers . . . . . . . . . . . : 10.2.0.27
                                       10.2.0.1
   NetBIOS over Tcpip. . . . . . . . : Enabled
...

Of course, it’s possible to run Powershell scripts like the next one which shows the system memory:

$strComputer = $Host
Clear
$RAM = WmiObject Win32_ComputerSystem
$MB = 1048576

"Installed Memory: " + [int]($RAM.TotalPhysicalMemory /$MB) + " MB"

The Python code to run that script is:

#!/usr/bin/env python

import winrm

ps_script = open('scripts/mem.ps1','r').read()
s = winrm.Session('10.2.0.42', auth=('the_username', 'the_password'))
r = s.run_ps(ps_script)
print r.status_code
print r.std_out
print r.std_err

and the output:

$ ./winrm_mem.py 
0
Installed Memory: 2217 MB

In the end it’s time to talk about how to create an Ansible Playbook to deploy anything in a Windows machine. As always the first thing that we need is a hosts file. In the next example there are several ansible variables needed to run Ansible Windows modules on WinRM, all of them are self-explanatory:

[all]
10.2.0.42

[all:vars]
ansible_ssh_user=the_username
ansible_ssh_pass=the_password
ansible_ssh_port=5985 #winrm (non-ssl) port
ansible_connection=winrm

The first basic example could be a simple playbook that runs the ‘ipconfig’ command and registers the output in an Ansible variable to be showed later like a debug information:

- name: test raw module
  hosts: all
  tasks:
    - name: run ipconfig
      raw: ipconfig
      register: ipconfig
    - debug: var=ipconfig

The command and the output to run latest example:

$ ansible-playbook -i hosts ipconfig.yml 

PLAY [test raw module] ******************************************************** 

GATHERING FACTS *************************************************************** 
ok: [10.2.0.42]

TASK: [run ipconfig] ********************************************************** 
ok: [10.2.0.42]

TASK: [debug var=ipconfig] **************************************************** 
ok: [10.2.0.42] => {
    "ipconfig": {
        "invocation": {
            "module_args": "ipconfig", 
            "module_name": "raw"
        }, 
        "rc": 0, 
        "stderr": "", 
        "stdout": "\r\nWindows IP Configuration\r\n\r\n\r\nEthernet adapter GigaBit 
...
        ]
    }
}

PLAY RECAP ******************************************************************** 
10.2.0.42                  : ok=3    changed=0    unreachable=0    failed=0 

As always Ansible have several modules, not only the ‘raw’ module. I committed two examples in my Github account using a module to download URLs and another one that runs Powershell scripts.

My examples are done using Ansible 1.8.2 installed in a Fedora 20. But main problems I’ve found are configuring Windows 7 to accept WinRM connections. Next I attach some references that helped me a lot:

If you want to use my tests code you can connect to my Github: Basic Ansible playbooks for Windows.

Jan 21

Using Ansible like library programming in Python

Reading time: 2 – 4 minutes

Ansible is a very powerful tool. Using playbooks, something like a cookbook, is very easy to automate maintenance tasks of systems. I used Puppet and other tools like that but IMHO Ansible is the best one.

In some cases you need to manage dynamic systems and take into advantage of Ansible like a Python library is a very good complement for your scripts. This is my last requirement and because of that I decided to share some simple Python snippets that help you to understand how to use Ansible as a Python library.

Firstly an example about how to call an Ansible module with just one host in the inventory (test_modules.py):

#!/usr/bin/python 
import ansible.runner
import ansible.playbook
import ansible.inventory
from ansible import callbacks
from ansible import utils
import json

# the fastest way to set up the inventory

# hosts list
hosts = ["10.11.12.66"]
# set up the inventory, if no group is defined then 'all' group is used by default
example_inventory = ansible.inventory.Inventory(hosts)

pm = ansible.runner.Runner(
    module_name = 'command',
    module_args = 'uname -a',
    timeout = 5,
    inventory = example_inventory,
    subset = 'all' # name of the hosts group 
    )

out = pm.run()

print json.dumps(out, sort_keys=True, indent=4, separators=(',', ': '))

As a second example, we’re going to use a simple Ansible Playbook with that code (test.yml):

- hosts: sample_group_name
  tasks:
    - name: just an uname
      command: uname -a

The Python code which uses that playbook is (test_playbook.py):

#!/usr/bin/python 
import ansible.runner
import ansible.playbook
import ansible.inventory
from ansible import callbacks
from ansible import utils
import json

### setting up the inventory

## first of all, set up a host (or more)
example_host = ansible.inventory.host.Host(
    name = '10.11.12.66',
    port = 22
    )
# with its variables to modify the playbook
example_host.set_variable( 'var', 'foo')

## secondly set up the group where the host(s) has to be added
example_group = ansible.inventory.group.Group(
    name = 'sample_group_name'
    )
example_group.add_host(example_host)

## the last step is set up the invetory itself
example_inventory = ansible.inventory.Inventory()
example_inventory.add_group(example_group)
example_inventory.subset('sample_group_name')

# setting callbacks
stats = callbacks.AggregateStats()
playbook_cb = callbacks.PlaybookCallbacks(verbose=utils.VERBOSITY)
runner_cb = callbacks.PlaybookRunnerCallbacks(stats, verbose=utils.VERBOSITY)

# creating the playbook instance to run, based on "test.yml" file
pb = ansible.playbook.PlayBook(
    playbook = "test.yml",
    stats = stats,
    callbacks = playbook_cb,
    runner_callbacks = runner_cb,
    inventory = example_inventory,
    check=True
    )

# running the playbook
pr = pb.run()  

# print the summary of results for each host
print json.dumps(pr, sort_keys=True, indent=4, separators=(',', ': '))

If you want to download example files you can go to my github account: github.com/oriolrius/programming-ansible-basics

I hope it was useful for you.

Dec 01

Short MIIMETIQ definition

Reading time: 2 – 3 minutes

M2MCF and MIIMETIQ

Last months in M2M Cloud Factory we have been working on MIIMETIQ. Last weeks I’ve been thinking about how to define MIIMETIQ shortly and this is my definition, please tell if you can understand something. Of course, you have to know we’re focused in Internet of Things and M2M market.

  • MIIMETIQ is an IoT/M2M framework, so this is the first step to setup to develop your vertical solution.
  • Framework: With a well defined architecture a framework is a set of functions ready to create any application. Everything else is open and adaptable.
  • MIIMETIQ architecture is service oriented and it uses AMQP as a message broker to connect the services.
  • MIIMETIQ has several modules, we define a module as a set of services. Basicly MIIMETIQ have 5 modules:
    • Identity Manager: manage users, groups, roles and all kind of entities the project needs and its security.
    • Assets Manager: a data model manager, the integrator creates the business logics and data models here.
    • Distribution System: this is a set of agnostitc connectivity layers to different types of devices.
    • A E N M: several time series and other signals flow through the AMQP, this data are events and using rules those events could be converted in alarms and some alarms have to be notified to proper services, systems or people.
    • Control Panel UI: this is an administration dashboard, in form of a UI to setup and monitor the most common uses of MIIMETIQ.
  • Using those modules usually the integrators create their own user interface to satisfy customer requiremests. In M2MCF we create those UI using ADUX (Advanced Development User Experience).
  • After configuring MIIMETIQ the integrator has 2 customized APIs to connect their code with MIIMETIQ. One of them is an API REST and another one is AMQP.
  • Finally everything inside MIIMETIQ could be customized, because the flexibility is very important when you have an horizontal solution.
Oct 21

OpenAM: some ssoadm commands for reference

Reading time: 3 – 4 minutes

OpenAM is as much powerful as complicated sometimes. In this case I spent a lot of time understanding how to set simple settings because of that I decide to take note about that in this blog entry.

First of all don’t forget to set the environment variables and go to ssoadm path:

export JAVA_HOME="/usr/lib/jvm/java-6-openjdk-amd64/jre"
export CLASSPATH="/var/lib/tomcat7/webapps/openam/WEB-INF/lib/policy-plugins.jar::/var/lib/tomcat7/webapps/openam/WEB-INF/lib/openam-core-11.0.0.jar"

cd /opt/openam/ssoadmin/openam/bin

Getting the list of user identities:

./ssoadm list-identities -u amadmin -f /tmp/oam.pwd -e / -t User -x "*"

anonymous (id=anonymous,ou=user,dc=openam)
demo (id=demo,ou=user,dc=openam)
serviceusername (id=serviceusername,ou=user,dc=openam)
amAdmin (id=amAdmin,ou=user,dc=openam)
Search of Identities of type User in realm, / succeeded.

another useful query would be:

./ssoadm list-identities -u amadmin -f /tmp/oam.pwd -e / -t Role -x "*"

No plug-ins configured for this operation

But as you can see it doesn’t work and I don’t know how to solve it.

Taking a look to GUI get to identities list with: Access Control > / (Top Level Realm) > Privileges

In this webpage you have a list of role identities, in my case I have only this: “All Authenticated Users”. Inside this identity I can set different privileges:

  • REST calls for Policy Evaluation (EntitlementRestAccess)
  • Read and write access to all log files (LogAdmin)
  • REST calls for searching entitlements (PrivilegeRestReadAccess)
  • Read access to all log files (LogRead)
  • Read and write access to all federation metadata configurations (FederationAdmin)
  • Read and write access only for policy properties (PolicyAdmin)
  • Read and write access to all configured Agents (AgentAdmin)
  • Read and write access to all realm and policy properties (RealmAdmin)
  • REST calls for managing entitlements (PrivilegeRestAccess)
  • Write access to all log files (LogWrite)

If we want to remove a privilege:

root@vm:/opt/openam/ssoadmin/openam/bin# ./ssoadm remove-privileges -u amAdmin -f /tmp/oam.pwd -e / -g EntitlementRestAccess -i "All Authenticated Users" -t role

Privileges were removed from identity, All Authenticated Users of type, role in realm, /.

or adding a privilege:

root@vm:/opt/openam/ssoadmin/openam/bin# ./ssoadm add-privileges -u amAdmin -f /tmp/oam.pwd -e / -g EntitlementRestAccess -i "All Authenticated Users" -t role

Talking about policies, exporting:

./ssoadm list-policies -u amadmin -f /tmp/oam.pwd -e / -o /tmp/policies.xml

and if we want to import the policies:

./ssoadm create-policies -u amAdmin -f /tmp/oam.pwd -e / --xmlfile /tmp/policies.xml

creating a user:

./ssoadm create-identity -u amadmin -f /tmp/oam.pwd  -e / -i serviceusername -t User --attributevalues "userpassword=servicepassword"

Useful references:

Jul 08

sslsnoop – hacking OpenSSH

Reading time: < 1 minute Using sslsnoop you can dump SSH keys used in a session and decode ciphered traffic. Supported algorithms are: aes128-ctr, aes192-ctr, aes256-ctr, blowfish-cbc, cast128-cbc. Basic sslsnoop information:

 $ sudo sslsnoop # try ssh, sshd and ssh-agent… for various things
 $ sudo sslsnoop-openssh live `pgrep ssh` # dumps SSH decrypted traffic in outputs/
 $ sudo sslsnoop-openssh offline –help # dumps SSH decrypted traffic in outputs/ from a pcap file
 $ sudo sslsnoop-openssl `pgrep ssh-agent` # dumps RSA and DSA keys

Take a look into the project in sslsnoop github page.

Jun 30

Enabling linux kernel to open LOTS of concurrent connections

Reading time: < 1 minute Just a small recipe about how to enable linux kernel to open tons of concurrent connections. Really simple and useful post entry.

echo “10152 65535″ > /proc/sys/net/ipv4/ip_local_port_range
sysctl -w fs.file-max=128000
sysctl -w net.ipv4.tcp_keepalive_time=300
sysctl -w net.core.somaxconn=250000
sysctl -w net.ipv4.tcp_max_syn_backlog=2500
sysctl -w net.core.netdev_max_backlog=2500
ulimit -n 10240
Jan 24

Conferència: La revolució dels mini-PC: Raspberry PI, Arduino i més

Reading time: 1 – 2 minutes

Ahir al vespre vaig fer una conferència a la FIB (Facultat d’Informàtica de Barcelona) dins de la UPC (Universitat Politècnica de Catalunya). En aquesta xerra vaig estar explicant què és i en que es diferència Arduino i Raspberry PI. A més de presentar tot un conjunt de solucions alternatives i experiències en el tema.

En aquest enllaç podeu trobar les transparències de:  La revolució dels mini-PC: Raspberry PI, Arduino i més. i el video el teniu disponible al servidor de la FIB.

Ara també teniu disponible el video a youtube:

i podeu veure les transparències des d’aquest mateix post:

Espero els vostres feedbacks als comentaris, desitjo que ús sigui útil.

Oct 04

Recording linux desktop and audio

Reading time: < 1 minute I have two full-HD displays as a desktop and I want to record the second one of them while I record mic too. The output format of the record has to be MKV with h264 as video codec and AAC as audio codec. After some tests with VLC and FFMPEG finally I get the solution with this command:

ffmpeg -f alsa -ac 2 -i default -f x11grab -r 15 -s 1920×1080+0+0 -i :0.0+1920,0 \
       -acodec pcm_s16le -vcodec libx264  -preset ultrafast  -threads 0 -y Test.mkv

when I finish recording the clip I have to convert the audio channel because if I try to convert the audio format while I’m recording the audio channel I have delays or sync problems with video channel.

ffmpeg -i Test.mkv -map 0:0 -map 0:1 \
       -c:v copy \
       -c:a:1 libfdk_aac -profile:a aac_he_v2 -b:a:1 96k \
       output.mkv
Sep 30

Hello World using ‘kombu’ library and python

This entry is part 4 of 4 in the series AMQP and RabbitMQ

Reading time: 4 – 7 minutes

Some times schemas and snippets don’t need large descriptions. If you think this is not enough in this case tell me and I’m going to add explanations.

Using a python library called kombu as an abstraction to talk with AMQP broker we are going to develop different message routes setting each type of Exchange. As a backend I used RabbitMQ with default configuration.

AMQP schema using an exchange of type direct

kombu-direct

Queue definition:

from kombu import Exchange, Queue

task_exchange = Exchange("msgs", type="direct")
queue_msg_1 = Queue("messages_1", task_exchange, routing_key = 'message_1')
queue_msg_2 = Queue("messages_2", task_exchange, routing_key = 'message_2')

The producer:

from __future__ import with_statement
from queues import task_exchange

from kombu.common import maybe_declare
from kombu.pools import producers


if __name__ == "__main__":
    from kombu import BrokerConnection

    connection = BrokerConnection("amqp://guest:guest@localhost:5672//")

    with producers[connection].acquire(block=True) as producer:
        maybe_declare(task_exchange, producer.channel)
        
        payload = {"type": "handshake", "content": "hello #1"}
        producer.publish(payload, exchange = 'msgs', serializer="pickle", routing_key = 'message_1')
        
        payload = {"type": "handshake", "content": "hello #2"}
        producer.publish(payload, exchange = 'msgs', serializer="pickle", routing_key = 'message_2')

One consumer:

from queues import queue_msg_1
from kombu.mixins import ConsumerMixin

class C(ConsumerMixin):
    def __init__(self, connection):
        self.connection = connection
        return
    
    def get_consumers(self, Consumer, channel):
        return [Consumer( queue_msg_1, callbacks = [ self.on_message ])]
    
    def on_message(self, body, message):
        print ("RECEIVED MSG - body: %r" % (body,))
        print ("RECEIVED MSG - message: %r" % (message,))
        message.ack()
        return
    

if __name__ == "__main__":
    from kombu import BrokerConnection
    from kombu.utils.debug import setup_logging
    
    setup_logging(loglevel="DEBUG")

    with BrokerConnection("amqp://guest:guest@localhost:5672//") as connection:
        try:
            C(connection).run()
        except KeyboardInterrupt:
            print("bye bye")


AMQP schema using an exchange of type fanout

kombu-fanout

Queue definition:

from kombu import Exchange, Queue

task_exchange = Exchange("ce", type="fanout")
queue_events_db = Queue("events.db", task_exchange)
queue_events_notify = Queue("events.notify", task_exchange)


The producer:

from __future__ import with_statement
from queues import task_exchange

from kombu.common import maybe_declare
from kombu.pools import producers


if __name__ == "__main__":
    from kombu import BrokerConnection

    connection = BrokerConnection("amqp://guest:guest@localhost:5672//")

    with producers[connection].acquire(block=True) as producer:
        maybe_declare(task_exchange, producer.channel)
        
        payload = {"operation": "create", "content": "the object"}
        producer.publish(payload, exchange = 'ce', serializer="pickle", routing_key = 'user.write')

        payload = {"operation": "update", "content": "updated fields", "id": "id of the object"}
        producer.publish(payload, exchange = 'ce', serializer="pickle", routing_key = 'user.write')

One consumer:

from queues import queue_events_db
from kombu.mixins import ConsumerMixin

class C(ConsumerMixin):
    def __init__(self, connection):
        self.connection = connection
        return
    
    def get_consumers(self, Consumer, channel):
        return [Consumer( queue_events_db, callbacks = [self.on_message])]
    
    def on_message(self, body, message):
        print ("save_db: RECEIVED MSG - body: %r" % (body,))
        print ("save_db: RECEIVED MSG - message: %r" % (message,))
        message.ack()
        return


if __name__ == "__main__":
    from kombu import BrokerConnection
    from kombu.utils.debug import setup_logging
    
    setup_logging(loglevel="DEBUG")

    with BrokerConnection("amqp://guest:guest@localhost:5672//") as connection:
        try:
            C(connection).run()
        except KeyboardInterrupt:
            print("bye bye")


AMQP schema using an exchange of type topic

kombu-topic

Queue definition:

from kombu import Exchange, Queue

task_exchange = Exchange("user", type="topic")
queue_user_write = Queue("user.write", task_exchange, routing_key = 'user.write')
queue_user_read = Queue("user.read", task_exchange, routing_key = 'user.read')
queue_notify = Queue("notify", task_exchange, routing_key = 'user.#')


The producer:

from __future__ import with_statement
from queues import task_exchange

from kombu.common import maybe_declare
from kombu.pools import producers


if __name__ == "__main__":
    from kombu import BrokerConnection

    connection = BrokerConnection("amqp://guest:guest@localhost:5672//")

    with producers[connection].acquire(block=True) as producer:
        maybe_declare(task_exchange, producer.channel)
        
        payload = {"operation": "create", "content": "the object"}
        producer.publish(payload, exchange = 'user', serializer="pickle", routing_key = 'user.write')

        payload = {"operation": "update", "content": "updated fields", "id": "id of the object"}
        producer.publish(payload, exchange = 'user', serializer="pickle", routing_key = 'user.write')

        payload = {"operation": "delete", "id": "id of the object"}
        producer.publish(payload, exchange = 'user', serializer="pickle", routing_key = 'user.write')

        payload = {"operation": "read", "id": "id of the object"}
        producer.publish(payload, exchange = 'user', serializer="pickle", routing_key = 'user.read')

One consumer:

from queues import queue_events_db
from kombu.mixins import ConsumerMixin

class C(ConsumerMixin):
    def __init__(self, connection):
        self.connection = connection
        return
    
    def get_consumers(self, Consumer, channel):
        return [Consumer( queue_events_db, callbacks = [self.on_message])]
    
    def on_message(self, body, message):
        print ("save_db: RECEIVED MSG - body: %r" % (body,))
        print ("save_db: RECEIVED MSG - message: %r" % (message,))
        message.ack()
        return


if __name__ == "__main__":
    from kombu import BrokerConnection
    from kombu.utils.debug import setup_logging
    
    setup_logging(loglevel="DEBUG")

    with BrokerConnection("amqp://guest:guest@localhost:5672//") as connection:
        try:
            C(connection).run()
        except KeyboardInterrupt:
            print("bye bye")


Simple queues

Kombu implements SimpleQueue and SimpleBuffer as simple solution for queues with exchange of type ‘direct’, with the same exchange name, routing key and queue name.

Pusher:

from kombu import BrokerConnection

connection = BrokerConnection("amqp://guest:guest@localhost:5672//")
queue = connection.SimpleQueue("logs")

payload = { "severity":"info", "message":"this is just a log", "ts":"2013/09/30T15:10:23" }
queue.put(payload, serializer='pickle')

queue.close()

Getter:

from kombu import BrokerConnection
from Queue import  Empty

connection = BrokerConnection("amqp://guest:guest@localhost:5672//")

queue = connection.SimpleQueue("logs")

while 1:
    try:
        message = queue.get(block=True, timeout=1)
        print message.payload
        message.ack()
    except Empty:
        pass
    except KeyboardInterrupt:
        break
    print message

queue.close()

The files

Download all example files: kombu-tests.tar.gz