Lajos Veres’ page

Setting up Devops Weekly newsletter’s archive

2021-02-28T00:00:00+00:00

I am a subscriber of the Devops weekly newsletter. I was wondering a few weeks ago whether it had any web-based archive or not. It would have been more convenient for me to read it on the web than in my mailer. I asked its author whether it had any public archives, and also if he would mind if I created one. He confirmed that it had not had and he gave its approval.

I am a subscriber since 2015, so I have a few hundred issues. Unfortunately not all of them, but enough to start to build something meaningful.

My goal was to build something simple based on technologies that I knew or I was relatively confident that they were not going to need lots of time to learn. I wanted to minimize the costs, but also wanted something stable.

I used previously GitHub pages with Jekyll. GitHub pages offers you free git-backed hosting. So it comes implicitly with version tracking and backups. Also, anybody can open Pull Requests to a public repository so potentially others could contribute easily later. I thought the best is to keep the project open anyway so I registered a new GitHub organisation and I created a repository for the project. GitHub pages automatically recognises if a repository is named as your <account>.github.io and it starts to serve its content under the same domain. devopsweeklyarchive.github.io in our case.

Jekyll is a ruby-based static site generator templating engine. At least from this project’s point of view, these are its most relevant features. Also, it has some nice free templates. I used previously Minimal Mistakes which is one of the most popular. It has multiple options for how you can use it in a project, but this starter template seemed to be the simplest. So I recreated the project repository from this template.

Configuring Minimal Mistakes is quite straight-forward. Mainly it means filling its _config.yml. Which is quite well documented.

...
title: MM
email:
description: >- # this means to ignore newlines until "baseurl:"
  Write an awesome description for your new site here. You can edit this
  line in _config.yml. It will appear in your document head meta (for
  Google search results) and in your feed.xml site description.
twitter_username: username
github_username: username
minimal_mistakes_skin: default
search: true

# Build settings
markdown: kramdown
remote_theme: mmistakes/minimal-mistakes
# Outputting
permalink: /:categories/:title/
paginate: 5 # amount of posts to show
paginate_path: /page:num/
timezone: # https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
...

Also, I had to delete the example posts from the _posts folder. Essentially this is all you need to build an empty site.

To test it locally, you should run this command:

$ bundle exec jekyll serve --watch --incremental

Its output looks like this:

Configuration file: /home/vlajos/devopsweeklyarchive.github.io/_config.yml
            Source: /home/vlajos/devopsweeklyarchive.github.io
       Destination: /home/vlajos/devopsweeklyarchive.github.io/_site
 Incremental build: enabled
      Generating... 
      Remote Theme: Using theme mmistakes/minimal-mistakes
       Jekyll Feed: Generating feed for posts
                    done in 27.228 seconds.
/usr/lib/ruby/vendor_ruby/pathutil.rb:502: warning: Using the last argument as keyword parameters is deprecated
 Auto-regeneration: enabled for '/home/vlajos/devopsweeklyarchive.github.io'
    Server address: http://127.0.0.1:4000
  Server running... press ctrl-c to stop.

It serves the website on the mentioned url: http://127.0.0.1:4000.

My local ruby installation was not in the best shape, but reinstalling the related modules fixed its problems.

The next step was to convert the mails into Markdown files. I gathered them into a single mailbox first and ended to use a set of scripts:

split_big_mbox_to_individuals.sh

As its name says, this one is just splitting the big mailbox to single files/issues with formail.

cat bigmbox | formail -ds sh -c \
    'cat > mails/msg.$FILENO'
rm $(grep -rl MAILER-DAEMON mails/)

mbox_to_original.sh

is stripping down the unneeded bits from the mails. Most of the mail headers and the mailing list footers are quite pointless on the web. Also, it names the files based on the issue number extracted from the Subject line.

source_file=mails/$1
id=$(grep -e ^Subject $source_file |\
    sed \
        -e 's/.*#//g' \
        -e 's/\?=//g' \
)
destination_file=originals/$id.mail.txt

grep -e ^Subject -e ^Date \
    <$source_file \
    >$destination_file

echo >>$destination_file

sed -e '1,/^$/ d' -e '/^--$/,$d' \
    <$source_file \
    >>$destination_file

original_to_post.sh

is responsible for the main Markdown conversion:

# Extracts the issue number from the file name.
id=$(echo $input|sed -e 's/.mail.txt//g' -e 's/.*\///g')

# These ones construct date variables from the Date header.
date=$(grep Date: $input|sed -e 's/Date: //g')
day=$(date --date "$date" '+%Y-%m-%d')
sec=$(date --date "$date" '+%Y-%m-%dT%H:%M:%S%:z')

# We generate the post name from the issue's date and number:
filename=../devopsweeklyarchive.github.io/_posts/$day-$id.md

# The title comes from the subject with escaping the # character.
# The mails were stored originally in quoted-printable encoding,
# so I needed to convert them to plain text.
title=$(\
    perl -MMIME::QuotedPrint -pe '$_=MIME::QuotedPrint::decode($_);' \
    <$input|\
    sed -e '0,/^$/d' -e '/^$/,$d' -e 's/#/\\#/g'|\
    tr '\n' ' '\
)

# Then we put together the post's header section:

echo '---' >$filename
echo "title: $title" >>$filename
echo "date: $sec" >>$filename
echo '---' >>$filename

# And its body: (I have injected some line-breaks and
# comments to make it a bit more readable.
# Those make it syntactically wrong.)

# Convert Quoted printable
perl -MMIME::QuotedPrint -pe '$_=MIME::QuotedPrint::decode($_);' \
    <$input |\

# Remove header section
    tail +5|\

# Remove standard footer
    grep -v 'If you received this email directly then' |\

# Remove double line-breaks before links
    perl -0 -p -e 's/\n\n *(http)/\n$1/g' |\

# Add explicit line-break before links
    perl -0 -p -e 's/\n(http)/\n<br>$1/g' |\

# Convert links to Markdown links
    sed -e 's/\(https\?:\/\/[^ ]*\)/[\1](\1)/g' \
    >>$filename

With these scripts, I could populate the _posts folder with the “main” content. I also added a simple about page.

I uploaded this version since it had the main parts in place already.

Then I tweaked it a bit more.

I did not like the long URL https://devopsweeklyarchive.github.io/, so I have bought a bit shorter one: https://devopsweeklyarchive.com/. Fortunately, GitHub pages supports these alternative names, just a CNAME file has to be provided.

GitHub pages offers now HTTPS for the domains with alternative names, but I also wanted to add Cloudflare to the picture. You can say it is overkill, but still, this is a hobby project, I wanted to play a bit with that as well. Also if you can have DDOS protection and world-wide CDN for free, then why not? Anyway, it was still the same simplicity to configure it as a few years ago. Originally I wanted to buy the domain from them as well, but currently, they offer transfer only. Although this page sounds quite attractive.

And a few more cherries to the top of the cake.

I wanted to add the basic Google management tools so registered for Analytics and Search Console Tools. Connecting them with the Minimal Mistakes template was very simple, just these lines in the configuration:

...
google_site_verification: "Google site verification code"

analytics:
  provider: "google-gtag"
  google:
    tracking_id: "Analytics account ID"
    anonymize_ip: false # default
...

I wanted to create a site map as well since the site structure is very basic, but when I checked, Minimal Mistakes already generated that for me.

The last piece was to make the site search a bit smarter. Minimal Mistakes came with a default search engine: Lunr, but it searched only in the first 50 words and it was not very sophisticated.

So I choose to enable Algolia. Fortunately, it is fully supported both with Minimal Mistakes and Jekyll.

And then I reached the project’s most puzzling problem. Why do these 300 pages generate 10000+ objects in Algolia? I needed the help of Algolia’s support and I had to dig a bit in its internals to understand that all <p> tags are going to be indexed separately to be able to search them individually. They describe this process here. Also, the original newsletter was organised to have two line-breaks between the “body” of an article and the source link of the article. Which meant that Jekyll generated two <p> tags from them. Understanding this took a bit long, but then removing one of the line-breaks was quite simple.

Another tricky question is how to integrate the indexing step into a static system? Luckily, they have a nice solution to this problem. You can configure Travis to execute some steps triggered by Git pushes. Travis is a much smarter CI system than this, but to solve this particular problem, we do not need its other capabilities. Essentially I needed to copy their example Travis file to the new repository. So now we have smart typo-resistant search functionality.

Finally, the Devops weekly newsletter got its public archive.

Verifying webserver configuration with some basic tests

2015-09-16T00:00:00+00:00

We have a project to restructure our webservers, consolidate their configurations, consider using new technologies and implement best practices where it is possible. Which means lots of configuration changes. Our setup is not too complex. We have only a reverse proxy, 2 webservers (one of them is the reverse proxy), a firewall and we use sometimes a CDN for different contents. Essentially, we serve 4 sites via https, we have a couple of aliases, redirects, rewrite rules, and some header manipulation for browser-side caching, security, and compression.

But if we count the different types of requests it is roughly

4 (virtual hosts) *
3 (caching logics) *
2 (http redirects, https versions) *
4-5 (aspects of the response we want to verify)

And a couple of security-related rules.

So the number is much higher than we would like to test it manually after each configuration change…

We googled but we could not find any tool which would offer any assertion mechanism for testing HTTP headers. To be honest, I am a bit surprised, maybe we were not thorough enough.

Anyway, we decided to implement something basic for ourselves which fulfills our needs and it is not more complex than the minimum.

Finally, we picked wget and wrote a couple of trivial bash functions around it.

The main logic of the test “framework”:

#!/bin/bash
wget(){
  export LASTRESULT=$(/usr/bin/wget -S --no-check-certificate --header="Accept-encoding: gzip" --max-redirect=0 --header="Host: $2" $1://$TargetIp:$3$4 -O /dev/null 2>&1)
  export LASTCALL="IP:$TargetIp $1://$2:$3$4"
}
testShouldContain(){
  echo "$LASTRESULT" |grep -q "$1" || echo "$LASTCALL should contain: '$1'"
}
testShouldNotContain(){
  echo "$LASTRESULT" |grep -q "$1" && echo "$LASTCALL should not contain: '$1'"
}

So it is around 10 lines and nothing super-fancy. (And uses global variables… Erm. It could be improved.)

But let’s see it in action:

#!/bin/bash

export TargetIp=x.y.z.a
#export TargetIp=127.0.0.1

source test_framework.sh
wget_oursite(){
  wget https oursite 443 "$@"
}

# Host/port related redirects
wget http www.oursite.com 80 /
testShouldContain "HTTP/1.1 301 Moved Permanently"
testShouldContain 'Location: https://www.oursite.com/'
wget https oursite.com 443 /
testShouldContain "HTTP/1.1 301 Moved Permanently"
testShouldContain 'Location: https://www.oursite.com/'

# Geo-location redirect
wget_oursite /
testShouldContain "HTTP/1.1 301 Moved Permanently"
testShouldContain "Content-Type: text/html; charset=UTF-8"
testShouldContain "Location: /uk/"
testShouldContain "X-Frame-Options: sameorigin"
testShouldContain "X-Content-Type-Options: nosniff"
testShouldNotContain "Content-Encoding: gzip"
testShouldNotContain "Vary: Accept-Encoding"
testShouldNotContain "Pragma: no-cache"
testShouldNotContain "Last-Modified:"

# A PHP page gzipped, non-cached:
wget_oursite /uk/
testShouldContain "HTTP/1.1 200 OK"
testShouldContain "Content-Type: text/html; charset=UTF-8"
testShouldContain "Content-Encoding: gzip"
testShouldContain "Vary: Accept-Encoding"
testShouldContain "Set-Cookie: PHPSESSID"
testShouldContain "Pragma: no-cache"
testShouldContain "X-Frame-Options: sameorigin"
testShouldContain "X-Content-Type-Options: nosniff"
testShouldContain "Expires: .* 198[0-9]"
testShouldNotContain "Last-Modified:"

# Static asset, gzippable
wget_oursite /uploads/app.css
testShouldContain "HTTP/1.1 200 OK"
testShouldContain "Content-Type: text/css"
testShouldContain "Last-Modified:"
testShouldContain "Cache-Control: max-age=604800"
testShouldContain "Expires:"
testShouldContain "Content-Encoding: gzip"
testShouldContain "Vary: Accept-Encoding"
testShouldNotContain "Pragma: no-cache"
testShouldNotContain "X-Frame-Options: sameorigin"
testShouldNotContain "X-Content-Type-Options: nosniff"

# Static asset, non gzippable
wget_oursite /uploads/picture.jpg
testShouldContain "HTTP/1.1 200 OK"
testShouldContain "Content-Type: image/jpeg"
testShouldContain "Last-Modified:"
testShouldContain "Cache-Control: max-age=604800"
testShouldContain "Expires:"
testShouldContain "ETag:"
testShouldNotContain "Content-Encoding: gzip"
testShouldNotContain "Vary: Accept-Encoding"
testShouldNotContain "Pragma: no-cache"
testShouldNotContain "X-Frame-Options: sameorigin"
testShouldNotContain "X-Content-Type-Options: nosniff"

# random not-existing
wget_oursite /not-existing.html
testShouldContain "HTTP/1.1 404 Not Found"

# server-status
wget_oursite /server-status
testShouldContain "HTTP/1.1 403 Forbidden"

# .htaccess
wget_oursite /.htaccess
testShouldContain "HTTP/1.1 403 Forbidden"

So in around 2 hours, we implemented 107 basic tests to verify our different assets, security, gzip, proxy, and caching-related rules. Its runtime is around 2 secs. So our main goal has been achieved. If we change anything in the setup we can verify that with these rules practically instantly.

Some more random comments:

The responses are cached. (At least until the following one.)
The asserts work with regular expressions. (see Expires: check)
The wget arguments are not 100% trivial so maybe they need some explanation:
- -S: We started with --debug but it gave us too many unneeded lines. -S displays only the response headers.
- --no-check-certificate: Unfortunately the server is not yet accessible under its final DNS name, so the url’s host we are connecting to is different than the name in the certificates.
- --header="Accept-encoding: gzip": We also want to test compression.
- --max-redirect=0: By default, wget would follow redirects, but in our case, this is not required.
- --header="Host: $2": We want to test different virtual hosts and the current DNS settings point to the live system.
- $1://$TargetIp:$3$4: This is only the construction of the URL.
- -O /dev/null: Throw away the downloaded content.
- 2>&1: Merge wget’s standard error into its output.
We could group the asserts by creating higher-level functions. For example testCacheHeaders, testGzipped and so on. Maybe we will do this later.

Orchestrating a high traffic static page after the last minute

2015-08-17T00:00:00+00:00

Some background:

We have launched one of our new pages a couple of days ago: Perceptions of Perfection

The backend team was not very involved in the development of this new page because it seemed to be relatively simple and frontend-only. We knew that it meant to be viral but we had no clue that this would be viral…

When I arrived Monday morning at the office, they were waiting for me with the news that something was not OK, the whole system was slow. I checked my emails and saw that there were a couple of New Relic alerts too.

I logged in to our frontend web server and realised that something was really wrong. Even the ssh connections lagged. The system load and vmstat did not seem to be too bad and there were not any interesting error log entries. I checked the site. Both our static and generated pages were very slow. Which meant in this case more than 20-30 seconds of loading time.

I checked our Geckoboard and saw that the apdex scores were very wrong. I assumed that maybe a bot attempting to place automated orders or something similar. I checked the server logs whether there were any suspiciously high traffic ip addresses but found none. My colleagues had let me know that the static pages were also very slow. Which meant the bot assumption was false…

I continued the investigation with the server statuses. This is one of apache’s most useful and (I think) less-known features. And we saw our whole system on top of its head… Most of the processes were in state “G”. It can be translated to “Gracefully finishing”. But what did this mean exactly? It was not very obvious and it was not 100% sure but it looked like our proxy webserver failed to handle the connections and dropped them before draining their content properly. So something had to be wrong a step before the apaches. I realised that this had to be a nginx problem.

We also noticed that most of our requests were against the same page. It was a relatively simple AJAX endpoint that was used to display the expected delivery modes for our sites. After a few minutes thinking I realised that this page’s result changed only once a day… We could cache it and we could temporarily replace the whole call with the static result. We chose the second option as a quick fix and one of my colleagues started to implement a proper caching logic. This was our first victory. The distribution of the apache requests looked instantly closer to normal than before and the site gained some responsiveness. But it was still far from perfect.

So we returned to our previous point: the cause of “G”s. I wanted to verify what the situation was around our nginx. After some typos and troubles, I enabled the equivalent of mod_status on our nginx server using this page.

I checked the numbers and saw that we had 2000 connections and most of them were waiting… Hmm. 2000 is very close to 2048. I opened our nginx config and realised that we had only 1 worker_process… Erm. Our server had 4 cpu cores (although virtuals). And the worker_connections value was 2048…

I did a quick nginx tuning, increased the number of worker processes to 8, and enabled multi_accept. So with these new values:

worker_processes  8;
multi_accept on;

the whole system suddenly became more responsive. This was our second victory. Unfortunately, it did not help much longer than the previous one… We checked it with firebug and the connection seemed to be OK, but the page was still not stable. It was still lagging and loading very slowly.

However, the site became a bit quicker but a couple of pages gave 502 bad gateway. Classic. Something was wrong with the apaches. With removing the block, enabling more nginx throughput, apache became the bottleneck again. But why? Mod_status gave us the answer. Most of the connections were blocked. A simple apache restart fixed this bad gateway issue.

The whole system became slightly better but it was still far from optimal. The nginx stats looked OK right now, but something was even wrong. My next ideas were tcpdump and netstat. Maybe something was wrong around SYN handling? TCP backlog issue? netstat -s, netstat -taupn |grep -c SYN

I verified quickly but there were only a couple of connections in SYN_RECV. Fortunately netstat gave us more clues.

The number of waiting connections was high. After some reading, it was obvious that they were only keepalive connections, anyway decreasing anything which could use resources seemed to be a good idea, so we decreased the keepalive_timeout to 45 from 75. The number decreased as well, but the responsiveness did not change.

So return back to netstat. Listing, searching, filtering and grouping the netstat results and watching it for a couple of minutes, I realised that the Send-Q values were high, 30-40 for hundreds of connections. It looked like something saturated. After a quick google search, I found iftop, a very neat and simple tool to monitor our network interface’s usage. Without -n (disable hostname lookup) it is mostly useless as most of the network tools, but with -n it showed us that we had 100Mb traffic… Hmm… I wanted to verify whether our interface was 100Mb or maybe 1Gb but I did not find any working tool. So we assumed it was 100Mb… It was not a 100% safe bet but it seemed to be easier to fix it than verify properly. (This was the point when I cursed the third time who installed our www machine and forgot to enable munin… and myself for not having fixed this issue yet…) Anyway, with the knowledge that our network was saturated, I wanted to verify why.

I checked our apache logs and the page above. I did not believe the results. There were requests with 1.8M result… I asked the frontend team to optimise those images quickly and they reduced their size to the third of the original.

In the meantime, I started to bootstrap a rackspace CDN service. It was surprisingly easy. I had never worked with CDN services before. In a nutshell, they work as a reverse proxy. You add a name and the “original” server and it starts to serve the original server’s content on the new name. And you can add rules for caching. It would be perfect if the caching worked… Anyway, the frontend team changed the URLs quickly to the CDN’s and I started to debug why it was not caching.

With adding a simple GET parameter to an image request it was easy to separate the test requests from the other storming traffic. Wget’s --debug showed us the certificate… Hmm, Akamai… Useful information. I tried different caching rules but I got the same results. None of them helped. Finally, I added a rule to cache everything for a day. (Later I realised that the default cache settings were perfect. Images are still cached for a day.) I compared the requests, assuming maybe the CDN’s caching behaviour was influenced somehow by the original request’s cache controlling headers. There were two interesting things: the original requests could be cached for 1 week and there was a Vary: Accept-Encoding. What was this? As a conclusion, it was obvious that the caching rule cannot be inherited by the original server’s rules. 1 week could not be interpreted as nothing. But what is this Vary: ? I remembered it should help cache servers to differentiate between requests and cache them separately based on the headers defined by Vary. This should not have been a problem. But why it was there? Anyway, simplicity sounded better and because this was our only remaining clue, I googled for it. Success! Or at least one more hint. Akamai did not like that Vary: header in other cases. Maybe we had the same problem. Anyway, we did not need these headers for our images. They could not be compressed anyway. But our whole directory had this Vary: header. I narrowed down its scope with a location to match only to the javascript and css files. And finally, the CDN started to serve clients from its cache and traffic dropped from 100Mbit to 5-8Mbit on the origin server. The site became fully responsive. We had few thousands of visitors at the same time. Happy End.

Until the end of the day, we had 120k shares, 9k likes, and 4k tweets.

Testing an alert was more complex than it seemed to be…

2015-07-21T00:00:00+00:00

Currently, we are working on a big refactoring project. We are rewriting a big chunk of legacy javascript with Angular.js. We also write end-to-end tests with protractor to keep our system more robust.

My colleagues used UI Bootstrap module’s alert function to display different things to our clients. And they wanted to test if the alert is displayed properly.

The first attempts returned with a strange error message:

10:55:37.099 WARN - Exception thrown org.openqa.selenium.TimeoutException: asynchronous script timeout: result was not received in 11 seconds

Essentially, the test clicked a button and wanted to check if there is an alert or not. It was very strange that - based on the logs - the click happened properly, but if we removed anything after the click() action the timeout remained still there. It took a bit of time to understand what happened in the background.

Unfortunately the alert’s internal dismiss functionality has been implemented with Angular’s $timeout service which increments $browser’s outstanding request count and protractor with ignoreSynchronization=false waits correctly until there are any outstanding requests. This means if you do not disable the synchronization, you cannot see the opened alert, because protractor suspends execution until it is visible.

So in our case, the click() opened the alert, which started the $timeout, and protractor suspended the execution and then it timed out.

This link helped us a lot to understand the situation:

If you want to test any $timeout based functionality you have to turn on ignoreSynchronization which is basically a hack for non-Angular apps.

There they suggest to use $interval because it fits better to this case.

As a work around currently we turned on ignoreSynchronization, but for a proper solution we implemented the suggested $interval transition and sent a Pull Request.

It seems that there are different opinions regarding this $timeout vs $interval debate. (See the comments in the PR.)

Jenkins, Browserstack integration: Link jenkins builds with Browserstack sessions

2015-07-02T00:00:00+00:00

We have a couple of Jenkins tasks for running tests with Browserstack. One suite contains 5-6 tests and 4 platforms configured with Jenkins’ matrix plugin. (We plan to increase both dimensions.)

If any of the tests fail and we want to investigate what happened we have to open a Browserstack window, navigate to the given build page and find on the page the given session. This can be painful if the suite is still running because Browserstack is continuously refreshing the page. When the tests are done at least the page is stable, so you can search with the browser’s find function. If the test had not run amongst the first ten of the given suite you have to click one more.

This is not so bad if you do it once or twice a day. Approximately 30 seconds and 4-5 clicks. But if you develop intensively and the tests are not stable yet this could take 10% of your time maybe even more.

Therefore we wanted to automate somehow this process. It did not seem to be too complex, Jenkins already communicates with Browserstack, though it does indirectly, but maybe we could send back the session’s link to it.

We asked Browserstack’s support. They told us that we could access the remote session’s id and with that id, we could also get the session’s url using Browserstack’s REST API.

Getting the webdriver session’s data:

    browser.driver.session_.then(function(sessionData) {
        console.log(sessionData);       // All session data
        console.log(sessionData.id_);   // The webdriver session id
    });

And we can call the REST API using this id on the end-point:

  var link= 'https://www.browserstack.com/automate/sessions/' + sessionData.id_ + '.json';

Unfortunately, this second link is protected with HTTP-Auth. But the expected login/password here is not the API credentials, but the web user’s one. So we have to allow Jenkins to use somehow one of our web user’s credentials.

The final code for this:

browser.driver.session_.then(function(sessionData) {
  require('https').request({
    host: 'www.browserstack.com',
    port: 443,
    path: '/automate/sessions/' + sessionData.id_ + '.json',
    method: 'GET',
    auth: 'login:password'
  },function(res){
    res.setEncoding('utf8');
    res.on('data', function(d) {
      var sessionUrl = JSON.parse(d).automation_session.browser_url;
      console.log('Browserstack session url: ' + sessionUrl);
    });
  }).end();
});

In our case, we displayed the capabilities as well like in the first snippet and we have an exception when we test locally instead of Browserstack and when the password is accessed somehow differently. So it is a bit more complex but this is the main concept.

You can place this snippet anywhere in your tests. If you want to run it for every test probably the best place is a centralised code. We added this to our protractor.conf’s onprepare section.

After implementing this we have got a line in our logs like this: Browserstack session url: https://www.browserstack.com/automate/builds/123.../sessions/456...

When we started to develop this function we did not know yet how we could use this link in Jenkins, so originally this was intended to be only a temporary verification step. Fortunately Jenkins has a Description Setter Plugin which can run regular expression against the build log and it can use their results customise the given build’s description.

You have to install the plugin first with these steps:

Manage Jenkins
Manage Plugins
Available
Description setter plugin

Then you have to configure it for all the tasks where you want to use it:

Open the task’s configuration page.
Scroll down to the Add post-build action’s and add the plugin’s call.
We use with the following settings:
- Regular expression: Browserstack session url: (.*)
- Description: <a href="\1">Browserstack session</a>
- Regular expression for failed builds: Browserstack session url: (.*)
- Description for failed builds: <a href="\1">Browserstack session</a>

The last two items appear when you open the advanced options. Without them, this feature is pretty useless for debugging… We tried to add target="_blank", but it was stripped out by Jenkins. Probably it could be added somehow…

And the result looks like this in the history:

#661 Jul 02, 2015 2:09 PM Browserstack session

Save our day with ghostscript to concatenate PDF files

2015-06-16T00:00:00+00:00

This issue started without any extra notice. Customer service notified us that labels were not uploaded. Essentially, we receive the labels from our logistic partners, we print them out and apply them to the parcels we send out. And there is also a small step in the system where we concatenate the label batches to bigger files to make the printing easier for our colleagues. Both the input and the output files are simple PDFs.

When we checked our logs we noticed that something had been changed with the original labels.

[... 11:01:06 2015] [error] [client 333.333.333.333] PHP Fatal error:
Uncaught exception 'Exception' with message 'This document (/tmp/label_55a780e22172f.pdf) probably uses a compression technique which is not supported by the free parser shipped with FPDI. (See https://www.setasign.com/fpdi-pdf-parser for more details)' in
...lib/FPDI-1.5.2/pdf_parser.php:329
Stack trace:
#0 ...lib/FPDI-1.5.2/pdf_parser.php(202): pdf_parser->_readXref(Array, 87345)
#1 ...lib/FPDI-1.5.2/fpdi_pdf_parser.php(71): pdf_parser->__construct('/tmp/...')
#2 ...lib/FPDI-1.5.2/fpdi.php(128): fpdi_pdf_parser->__construct('/tmp/..')
#3 ...lib/FPDI-1.5.2/fpdi.php(108): FPDI->_getPdfParser('/tmp/...')
#4 ...php/utils/concatpdf.class.php(37): FPDI->setSourceFile('/tmp/...')
#5 ...php/utils/shipping/provider in ...lib/FPDI-1.5.2/pdf_parser.php on line 329

The error message was quite clear: the input PDF files’ format had been changed. We could verify it easily:

$ file label_55a780e22172f.pdf
label_55a780e22172f.pdf: PDF document, version 1.7

We checked the link the error message mentioned… They have a commercial add-on to open PDF files higher than 1.4.

Fortunately, I played with ghostscript a few days ago and remembered that it could manipulate PDF files efficiently. And Google gave us useful examples on stackoverflow.

We implemented the new code in less than half an hour. The main logic was very simple:

gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=$outfile $inputfiles

We gave it a try and it worked. We tried it with longer files and we noticed that a few labels had changed their orientation. 2-3 of 60 labels were landscape instead of portrait, though all the originals were portrait. And the problem appeared consistent with the same files.

We started to play with the switches… Ghostscript had much more options than we expected and its documentation is not really Google-friendly, but we found finally the AutoRotatePages flag.

Our first attempt was the -dAutoRotatePages=/All setting, but this uses some kind of majority decision for rotating the pages… We tried this with only one label which had been rotated unintentionally. It did not work as we could expect. It seemed that something like this had been enabled on page level. At least with this given file ghostscript did the same rotation with or without the flag.

Fortunately, our next attempt was successful. We added -dAutoRotatePages=/None Which worked properly for both the big batches and the single files.

gs -dAutoRotatePages=/None -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=$outfile $inputfiles

A 27 years old software saved our day.

Link to its version 1.n History

A lucky shot with strace to optimise database

2015-05-14T00:00:00+00:00

I received a task to improve our shipment tracking system. A part of the task was to poll more frequently our partner site’s remote statuses. Essentially, I had to change the two times a day cadence to 2 times an hour. We already had a cronjob to manage this task, but I was not very familiar with this sub-system.

I hoped a bit that maybe it is enough to reconfigure the job’s crontab entry. Anyway, I wanted to play safely so checked first how long the given tasks run. The result was really sad. The task worked on only four day’s data but ran for 75 minutes. The math was simple 75 minutes was too much.

As far as I knew the job polls only the remote systems and saves the result in the database.

We did not have too many profiling tools installed on the related machine, so I chose to use the simplest available tool: strace.

I ran the job in one terminal and started a strace in another.

The result was surpisingly conclusive, the character flow stopped for seconds at the following lines:

write(6, "\17\1\0\0\3\n            select\n       "..., 275) = 275

Strace’s -s switch helped to find the guilty query exactly. It was a very simple one:

select field from table where remote_id=?

The table received its well-deserved index in minutes and the next run of the job was only 14 minutes.