As every reader of this blog certainly know, I’m a big fan of Puppet, using it in production on Days of Wonder servers, up to the point I used to contribute regularly bug fixes and new features (not that I stopped, it’s just that my spare time is a scarce resource nowadays).

Still, I think there are some issues in term of scalability or resource consumption (CPU or memory), for which we can find some workarounds or even fixes. Those issues are not a symptom bad programming or bad design. No, most of the issues come either from ruby itself or some random library issues.

Let’s review the things I have been thinking about lately.

Memory consumption

This is by far one of the most seen issues both on the client side and the server side. I’ve mainly seen this problem on the client side, up to the point that most people recommend running puppetd as cronjobs, instead of being a long lived process.

Ruby allocator

All boils down to the ruby (at least the the MRI 1.8.x version) allocator. This is the part in the ruby interpreter that deals with memory allocations. Like in many dynamic languages, the allocator manages a memory pool that is called a heap. And like some other languages (among them Java), this heap can never shrink and always grows when more memory is needed. This is done this way because it is simpler and way faster. Usually applications ends using their nominal part of memory and no more memory has to be allocated by the kernel to the process, which gives faster applications.

The problem is that if the application needs transiently a high amount of memory that will be trashed a couple of millisecond after, the process will pay this penalty all its life, even though say 80% of the memory used by the process is free but not reclaimed by the OS.

And it’s even worst. The ruby interpreter when it grows the heap, instead of allocating bytes per bytes (which would be really slow) does this by chunk. The whole question is what is the proper size of a chunk?

In the default implementation of MRI 1.8.x, a chunk is the size of the previous heap times 1.8. That means at worst a ruby process might end up allocating 1.8 times more than what it really needs at a given time. (This is a gross simplification, read the code if you want to know more).

Yes but what happens in Puppet?

So how does it apply to puppetd?

It’s easy, puppetd uses memory for two things (beside maintaining some core data to be able to run):

1. the catalog (which contains all resources, along with all templates) as shipped by the puppetmaster (i.e. serialized) and live as ruby objects.
2. the content of the sourced files (one at a time, so it’s the biggest transmitted file that imposes it’s high watermark for puppetd). Of course this is still better than in 0.24 where the content was transmitted encoded in XMLRPC adding the penalty of escaping everything…

Hopefully, nobody distributes large files with Puppet If you’re tempted to do so, see below…

But again there’s more, as Peter Meier (known as duritong in the community) discovered a couple of month ago: when puppetd gets its catalog (which by the way is transmitted in json nowadays), it also stores it as a local cache to be able to run if it can’t contact the master for a subsequent run. This operation is done by unserializing the catalog from json to ruby live objects, and then serializing the laters to YAML. Beside the evident loss of time to do that on large catalog, YAML is a real memory hog. Peter’s experience showed that about 200MB of live memory his puppetd process was using came from this final serialization!

So I had the following idea: why not store the serialized version of the catalog (the json one) since we already have it in a serialized form when we receive it from the master (it’s a little bit more complex than that of course). This way no need to serialize it again in YAML. This is what ticket #2892 is all about. Luke is committed to have this enhancement in Rowlf, so there’s good hope!

Some puppet solutions?

So what can we do to help puppet not consume that many memory?

In theory we could play on several factors:

• Transmit smaller catalogs. For instance get rid of all those templates you love (ok that’s not a solution)
• Stream the serialization/deserialization with something like Yajl-Ruby
• Use another ruby interpreter with a better allocator (like for instance JRuby)
• Use a different constant for resizing the heap (ie replace this 1.8 by 1.0 or less on line 410 of gc.c). This can be done easily when using Rails machine GC patches or Ruby Enterprise Edition, in which case setting the environment variable RUBY_HEAP_SLOTS_GROWTH_FACTOR is enough. Check the documentation for more information.
• Stream the sourced file on the server and the client (this way only a small buffer is used, and the total size of the file is never allocated). This one is hard.

Note that the same issues apply to the master too (especially for the file serving part). But it’s usually easier to run a different ruby interpreter (like REE) on the master than on all your clients.

Streaming HTTP requests is promising but unfortunately would require large change to how Puppet deals with HTTP. Maybe it can be done only for file content requests… This is something I’ll definitely explore.

This file serving thing let me think about the following which I already discussed several time with Peter…

File serving offloading

One of the mission of the puppetmaster is to serve sourced file to its clients. We saw in the previous section that to do that the master has to read the file in memory. That’s one reason it is recommended to use a dedicated puppetmaster server to act as a pure fileserver.

But there’s a better way, provided you run puppet behind nginx or apache. Those two proxies are also static file servers: why not leverage what they do best to serve the sourced files and thus offload our puppetmaster?

This has some advantages:

• it frees lots of resources on the puppetmaster, so that they can serve more catalogs by unit time
• the job will be done faster and by using less resources. Those static servers have been created to spoon-feed our puppet clients…

In fact it was impossible in 0.24.x, but now that file content serving is RESTful it becomes trivial.

Of course offloading would give its best if your clients requires lots of sourced files that change often, or if you provision lots of new hosts at the same time because we’re offloading only content, not file metadata. File content is served only if the client hasn’t the file or the file checksum on the client is different.

An example is better than thousand words

Imagine we have a standard manifest layout with:

• some globally sourced files under /etc/puppet/files and
• some modules files under /etc/puppet/modules/<modulename>/files.

Here is what would be the nginx configuration for such scheme:

server {
    listen 8140;

    ssl                     on;
    ssl_session_timeout     5m;
    ssl_certificate         /var/lib/puppet/ssl/certs/master.pem;
    ssl_certificate_key     /var/lib/puppet/ssl/private_keys/master.pem;
    ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;
    ssl_crl                 /var/lib/puppet/ssl/ca/ca_crl.pem;
    ssl_verify_client       optional;

    root                    /etc/puppet;

    # those locations are for the "production" environment
    # update according to your configuration

    # serve static file for the [files] mountpoint
    location /production/file_content/files/ {
        # it is advisable to have some access rules here
        allow   172.16.0.0/16;
        deny    all;

        # make sure we serve everything
        # as raw
        types { }
        default_type application/x-raw;

        alias /etc/puppet/files/;
    }

    # serve modules files sections
    location ~ /production/file_content/[^/]+/files/ {
        # it is advisable to have some access rules here
        allow   172.16.0.0/16;
        deny    all;

        # make sure we serve everything
        # as raw
        types { }
        default_type application/x-raw;

        root /etc/puppet/modules;
        # rewrite /production/file_content/module/files/file.txt
        # to /module/file.text
        rewrite ^/production/file_content/([^/]+)/files/(.+)$  $1/$2 break;
    }

    # ask the puppetmaster for everything else
    location / {
        proxy_pass          http://puppet-production;
        proxy_redirect      off;
        proxy_set_header    Host             $host;
        proxy_set_header    X-Real-IP        $remote_addr;
        proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header    X-Client-Verify  $ssl_client_verify;
        proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
        proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
        proxy_buffer_size   16k;
        proxy_buffers       8 32k;
        proxy_busy_buffers_size    64k;
        proxy_temp_file_write_size 64k;
        proxy_read_timeout  65;
    }
}

EDIT: the above configuration was missing the only content-type that nginx can return for Puppet to be able to actually receive the file content (that is raw).

I leave as an exercise to the reader the apache configuration.

It would also be possible to write some ruby/sh/whatever to generate the nginx configuration from the puppet fileserver.conf file.

And that’s all folks, stay tuned for more Puppet (or even different) content.

Yes, I know… I released v0.7 less than a month ago. But this release was crippled by a crash that could happen at start or reload.

Changes

Bonus in this new version, brought to you by Tizoc:

• JSONP support
• Long awaited fix for X-Progress-ID to be the last parameter in the request parameter

If you wonder what JSONP is (as I did when I got the merge request), you can check the original blog post that lead to it.

To activate JSONP you need:

1. to use the upload_progress_jsonp_output in the progress probe location
2. declare the JSONP parameter with the upload_progress_jsonp_parameter

This version has been tested with 0.7.64 and 0.8.30.

How do you get it?

Easy, download the tarball from the nginx upload progress module github repository download section.

If you want to report a bug, please use the Github issue section.

I’m proud to announce the release of Nginx Upload Progress module v0.7

This version sees a crash fix and various new features implemented by Valery Kholodkov (the author of the famous Nginx Upload Module).

This version has been tested with Nginx 0.7.64.

Changes

• fixed segfault when uploads are aborted (thanks to Markus Doppelbauer for his bug report)
• session ID header name is now configurable (thanks to Valery Kholodkov)
• Added directive to format output as pure json (thanks to Valery Kholodkov)
• Added directive to format output with configurable template (thanks to Valery Kholodkov)
• Added directive to set a probe response content-type (thanks to Valery Kholodkov)
• Added upload status variables (needs a status patch) (thanks to Valery Kholodkov)

What’s now cool!

What is cool is that now with only one directive (upload_progress_json_output) the responses are sent in pure Json and not in javascript mix as it was before.

Another cool feature is the possibility to use templates to send progress information. That means with a simple configuration change nginx can now return XML:

upload_progress_content_type 'text/xml';
upload_progress_template starting '<upload><state>starting</state></upload>';
upload_progress_template uploading '<upload><state>uploading</state><size>$uploadprogress_length</size><uploaded>$uploadprogress_received</uploaded></upload>';
upload_progress_template done '<upload><state>done</state></upload>';
upload_progress_template error '<upload><state>error</state><code>$uploadprogress_status</code></upload>';

Refer to the README in the distribution for more information.

How do you get it?

Easy, download the tarball from the nginx upload progress module github repository download section.

How can I use it?

Normally you have to use your own client code to display the progress bar and contact nginx to get the progress information.

But some nice people have created various javascript libraries doing this for you:

• JQuery upload progress module

• Protoype upload progress module

Happy uploads!

As a Puppet Mongrel Nginx user, I’m really ashamed about the convoluted nginx configuration needed (two server blocks listening on different ports, you need to direct your clients CA interactions to the second port with –ca_port), and the lack of support of proper CRL verification.

If you are like me, then there is some hope in this blog post.

Last week-end, I did some intense Puppet hacking (certainly more news about this soon), and part of this work is two Nginx patch:

• The first one adds support for ssl_client_verify optional. In this mode nginx accepts a client without a certificate, and of course accepts a client as long as it verifies against the CA certificate.
• The second patch adds support for CRL PEM files (the one we usually deploy in Puppet).

Installation

First, download both patches:

• Support ssl_client_verify optional and $ssl_client_verify
• Add SSL CRL verifications

Then apply them to Nginx (tested on 0.7.59):

$ cd nginx-0.7.59
$ patch -p1 < ../0001-Support-ssl_client_verify-optional-and-ssl_client_v.patch
$ patch -p1 < ../0002-Add-SSL-CRL-verifications.patch

Then build Nginx as usual.

Usage

Here is a revised Puppet Nginx Mongrel configuration:

upstream puppet-production {
  server 127.0.0.1:18140;
  server 127.0.0.1:18141;
}

server {
  listen 8140;

  ssl                     on;
  ssl_session_timeout     5m;
  ssl_certificate         /var/lib/puppet/ssl/certs/puppetmaster.pem;
  ssl_certificate_key     /var/lib/puppet/ssl/private_keys/puppetmaster.pem;
  ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;
  ssl_ciphers             SSLv2:-LOW:-EXPORT:RC4+RSA;
  # allow authenticated and client without certs
  ssl_verify_client       optional;
  # obey to the Puppet CRL
  ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;

  root                    /var/tmp;

  location / {
    proxy_pass              http://puppet-production;
    proxy_redirect         off;
    proxy_set_header    Host             $host;
    proxy_set_header    X-Real-IP        $remote_addr;
    proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header    X-Client-Verify  $ssl_client_verify;
    proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
    proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
    proxy_read_timeout  65;
  }
}

Reload nginx, and enjoy

It’s been a long time since my last post… which just means I was really busy both privately, on the Puppet side and at work (I’ll talk about the Puppet side soon, for the private life you’re on the wrong blog :-)).

For a project I’m working on at Days of Wonder, I had to use the nginx secure link module. This module allows a client to access to the pointed resource only if the given MD5 HashMAC matches the arguments.

Nginx current secure link module

To use it, it’s as simple as:

1. have your protected resources in /var/www/protected
2. have your back-end generate the correct url (see below)
3. use the following nginx config

location /protected/ {
secure_link "this is my secret";
root /var/www/downloads;

if ($secure_link = "") {
return 403;
}

rewrite ^ /$secure_link break;
}

To generate an URL, use the following PHP snippet:

<?php

$prefix = "http://www.domain.com/protected";

$protected_resource = "my-super-secret-resource.jpg";

$secret = "this is my secret";

$hashmac = md5( $protected_resource . $secret );

$url = $prefix . "/" . $hashmac . "/" . $protected_resource;

?>

I want my protected URL to expire

But that wasn’t enough for our usage. We needed the url to expire automatically after some time. So I crafted a small patch against Nginx 0.7.59.

How does it work?

It just extends the nginx secure link module with a TTL. The time at which the resource expires is embedded in the url, and the HMAC. If the server finds that the current time is greater than the embedded time, then it denies access to the resource.

The timeout can’t be tampered as it is used in the HMAC.

Usage

The usage is the same as the current nginx secure link module, except:

1. you need to embed the timeout into the URL
2. you need to tell nginx about the TTL.

On the back-end site

You need to use the following (sorry only PHP) code:

define(URL_TIMEOUT, 3600) # one hour timeout
$prefix = "http://www.domain.com/protected";
$protected_resource = "my-super-secret-resource.jpg";
$secret = "this is my secret";
$time = pack('N', time() + URL_TIMEOUT);
$timeout = bin2hex($time);

$hashmac = md5( $protected_resource . $time . $secret );

$url = $prefix . "/" . $hashmac . $timeout . "/" . $protected_resource;

On Nginx side

location /protected/ {
secure_link "this is my secret";
secure_link_ttl on;
root /var/www/protected;

if ($secure_link = "") {
return 403;
}

rewrite ^ /$secure_link break;
}

Caveat

The server generating the url and hashmac and the one delivering the protected resource must have synchronized clocks.

There is no support. If it eats your server, then I or Days of Wonder can’t be

I want it!

It’s simple:

1. download the nginx secure link ttl patch
2. apply it to nginx-0.7.59 source tree (patch -p0 < nginx-secure-link-ttl.patch)
3. configure nginx with –with-http_secure_link_module
4. use and abuse