The setup

At a customer, we have a clustered Alfresco setup with docker containers. This is a high-availability setup with two nodes. Clients connect through an externally managed F5 loadbalancer, which then connects to one of our Haproxy loadbalancers.

Under normal conditions, Haproxy sends traffic to both Alfresco servers. The setup with two levels of loadbalancers is made because we want to be able to disable an Alfresco backend during deployments.

graph LR
    client --> f5-loadbalancer
    f5-loadbalancer --> haproxy1
    f5-loadbalancer --> haproxy2
    subgraph node1
        haproxy1 --> alfresco1
    end
    subgraph node2
        haproxy2 --> alfresco1
        haproxy1 --> alfresco2
        haproxy2 --> alfresco2
    end

Because Alfresco is running in docker containers, they don’t have fixed IP addresses that can be put in the Haproxy configuration. This is one of the reasons why we have a consul cluster running at each client. A helper process will grab the containers IP addresses from consul, then writes the configuration for haproxy and sends a SIGUSR2 to haproxy, which will cause it to reload its configuration.

A problem

One fateful night, Alfresco needs to be shut down for a couple of hours for a scheduled database maintenance operation. After this database maintenance, Alfresco is started up again. A manual sanity check indicates that both Alfresco nodes are back up and running and everything is fine.

However, not everything was fine. In the morning, we receive a phone call that requests to Alfresco are failing with HTTP 503 Service Unavailable. After restarting haproxy, these errors subdue.

Investigation

As the reader of this blog might already have noticed, I do not like it when things go badly and I don’t know what caused the issue. After all, the only way to prevent the same issue from wreaking havoc again is to find the root cause and to eliminate it.

Since we already restarted some services to recover as fast as possible, some amount of evidence is gone now. But thanks to Kibana, we still have a good amount of data to work with after the fact.

Finding the culprit

We make a graph of the haproxy access logs, where we graph HTTP 503 responses (red) and all other responses (green).

Since only some requests were answered with an error, we try to find out what’s wrong exactly by splitting the graphs based on which node the haproxy server runs on, and which backend haproxy is sending to.

These graphs look very strange in two ways:

1. Traffic appears to be only sent to one Alfresco node.
2. Traffic of the one haproxy server to one backend is returning a 503 error half of the time.

It is expected that traffic is only sent to one Alfresco node. Haproxy is configured to use sticky loadbalancing based on source IP. Since all traffic comes from the F5 loadbalancer, there is only one source IP, and everything is sent to the same (first) backend server.

However, that half of the requests to one haproxy server and backend are returning a 503 error is quite unexpected. We know that that haproxy server and that Alfresco backend are running on the same server, so network issues can be excluded as a cause.

Things that we already encountered before that return 503 errors are:

• Haproxy configuration is not updated when the Alfresco container is recreated and changed IP address, resulting in haproxy not being able to reach the Alfresco backend
• Docker overlay networking is broken, and containers can’t reach each other
• Alfresco healtcheck failed on both nodes, and both nodes were removed from the haproxy configuration

None of these cases explains why some requests are succeeding and others failing, and why the situation recovered after haproxy was restarted.

Into the failing haproxy container

Looking at the logs of that haproxy server in kibana showed something strange:

<134>Dec 20 05:50:56 haproxy[45085]: xxx.xxx.xxx.194:2133 [20/Dec/2019:05:50:56.643] internet-xxx.xxx.xxx.82 alfresco/alfresco-core0 0/0/1/123/129 200 63108 - - --NN 4/4/0/1/0 0/0 "GET /alfresco/ HTTP/1.1"
<134>Dec 20 05:50:56 haproxy[31628]: xxx.xxx.xxx.196:38412 [20/Dec/2019:05:50:49.099] internet-xxx.xxx.xxx.82 alfresco/alfresco-core0 0/5518/-1/-1/7523 503 212 - - SCNN 1/1/1/0/2 0/0 "GET /alfresco/ HTTP/1.1"

There are many lines similar to these, where about half of these requests succeed, and the other half fail.

One thing that we notice when looking at the log messages is that failures are consistently logged by haproxy[31628] and successful requests are logged by haproxy[45085]. Failing requests are also always coming from the same set of IPs and ports.

In normal circumstances, there should only be one haproxy worker process that handles requests, as nbproc is set to 1. The fact that 2 worker processes are running at the same time is an indication that something went wrong.

How haproxy reloads its configuration

Like many *nix daemons, haproxy implements a reload mechanism that allows loading a new configuration without stopping and starting the server. In haproxy, there are multiple ways the reload mechanism works, depending on the mode in which haproxy is started. We are using the master-worker mode, so I will only describe that mechanism here.

The reload starts when the master process receives a SIGUSR2 signal. The master process will then re-exeute itself with the -sf parameter, followed by PIDs of all its worker processes.

Next, the master process will start listening on the configured ports with SO_REUSEPORT, so it can bind in parallel with the existing old worker processes. New worker processes are started by the master process. They inherit the listening socket and start serving new requests.

Finally, the master process sends the SIGUSR1 signal to all the old workers (listed in the -sf parameter). The old workers will stop themselves once they have finished processing all existing connections.

sequenceDiagram
    activate Old worker
    User-->>Haproxy master: SIGUSR2
    activate Haproxy master
    Haproxy master ->> Haproxy master: Re-execute itself
    Note over Haproxy master: Start listening on ports
    Haproxy master ->> New worker: Execute new worker
    activate New worker
    Haproxy master --x Old worker: SIGUSR1
    Note over Old worker: Stop after processing all pending requests
    deactivate Old worker
    deactivate New worker

Why master-worker is problematic with multiple levels of loadbalancers

This is fine when users enter directly through haproxy. It does not take that long for a normal user to close their connection, as browsers will typically only keep a connection alive for a couple of minutes after the last request is made through that TCP connection.

However, in the case that there is another layer of loadbalancers in front, that loadbalancer can keep sending requests from different clients over the same TCP connection, keeping the process alive indefinitely, and thus keeps sending clients to a no longer existing backend IP. The F5 loadbalancer healthcheck does not detect this problem, because it is sent with a new TCP connection every time.

Solving the problem

An initial fix consisted of configuring hard-stop-after, to ensure that worker processes will exit after some time. This way, we still get all the benefits from a soft reload, not dropping all user connections immediately, but we limit the damage that is done when TCP connections are kept alive for a very long time and keep their connections to a stale worker process.

A couple of months later, we embarked on a mission to simplify our architecture and cut out unnecessary complexity.

We found that instead of using Consul, consul-template and haproxy reloads, we could configure haproxy to query Docker DNS directly, by making use of the server-template feature.

Being able to remove Consul and consul-template for loadbalancing made the stack simpler to configure and easier to understand for everyone. What previously consisted of 3 containers and a consul cluster is now just one container and only an overlay network.

Configuring haproxy to use Docker DNS is not that hard and works the same in every Docker container.

resolvers docker
  nameserver docker 127.0.0.11:53
  resolve_retries 3
  timeout resolve 1s
  timeout retry   1s
  hold other      10s
  hold refused    10s
  hold nx         10s
  hold timeout    10s
  hold valid      10s
  hold obsolete   10s

Instead of templating all the backend servers in the haproxy configuration file, we can configure them with one line:

backend alfresco
  server-template alfresco- 2 alfresco:8080 check resolvers docker init-addr libc,none

Discovery

On my laptop, I run Debian 10, with systemd-resolved for DNS. My VPN client hooks into systemd-resolved to add a private DNS server and local domains (including *.service.consul and *.node.consul) to the configuration. That makes it very easy to connect to internal services by name. Multiple instances of these services are deployed to achieve high availability, consul DNS will always direct me to a working instance.

Last week I upgraded my laptop from Debian 9 to 10. This also came with a change in defaults for systemd-resolved: DNSSEC is now set to allow-downgrade by default, which will send additional DNS queries.

After connecting to the VPN, I try to connect to a consul service. This fails because of a DNS resolution failure.

Manually trying to resolve the name with dig results in connection timeouts as well.

Then trying with systemd-resolve results in an other error message.

lars@lars-debian:~$ systemd-resolve xyz.service.consul
xyz.service.consul: resolve call failed: DNSSEC validation failed: no-signature

The logs of systemd-resolved are filled with these DNSSEC validation failures too. After a lot of retried attempts, systemd-resolved marks the DNS server as non-DNSSEC capable: Jul 18 14:31:22 lars-debian systemd-resolved[635]: Server 10.88.30.3 does not support DNSSEC, downgrading to non-DNSSEC mode.

Unfortunately, at this time the damage is already done.

Walking in circles

What happened? To check for DNSSEC support, systemd-resolved sends a DNS query for the DS record type of consul. to its configured recursor.

In this case, that is the dnsmasq server running at 10.88.10.3. dnsmasq receives the query, and sees that a specific resolver has been configured for consul.. It forwards the DNS query to the consul server at 10.88.10.2 to resolve it.

Consul is configured with a recursor, so DNS queries for other (not *.consul) domains would be forwarded to the same dnsmasq server.

However, the underlying DNS server implementation of consul delegates DS queries to the parent handler, which are the configured recursors.

The result is that a DS consul. query is sent from consul 10.88.10.3 to the dnsmasq server running at 10.88.10.2. Upon seeing a query for consul., dnsmasq forwards it back to consul, and round and round in circles it goes.

Amplification

Dnsmasq has a maximum number of concurrent pending queries, by default 150. When this limit is reached, it starts answering SERVFAIL to all new requests. This response starts making its way back up the stack of recursive DNS queries.

Upon receiving a SERVFAIL answer, dnsmasq will retry the DNS query once more. There is a stack of 150 pending queries that are waiting for a response of the previous query. Every time a slot becomes available, it is immediately occupied by a retry.

The result is that very little legitimate DNS queries are able to get through. The only way to resolve this loop is to break it by restarting either dnsmasq or consul.

Conclusion

Don’t create a loop in your DNS recursors’ configuration. In our case, nothing was querying consul directly anyways, since the dnsmasq server was configured the system resolver.

I reported an issue to consul, because either the documentation or the code is incorrect. This lead us to think that our configuration was okay.

It is common to install these dependencies in a virtualenv instead of installing them globally on the system. However, this would mean that to use this small commandline tool, I would first need to activate the virtualenv before running the command. When using and developing these utilities, I do not want to have to keep in mind to activate the correct virtualenv every time.

And I also do not want to globally install all libraries that the application depends on, because that may result in a dependency hell when two tools use different, incompatible versions of the same library.

A polyglot

While working on the ULYSSIS CTF, one of those challenges made me think about an interesting approach that embeds a shellscript at the start of a Python file.

The basic structure of such a file consists of starting with a shellscript inside a Python docstring, and then following it with actual Python code.

''''echo "Shell code here"
exit
'''
print("Python code here")

When run with a shell, '''' is interpreted as two strings, which does nothing at all. Shell scripts are somewhat executed line by line, so nothing after exit is interpreted and the shell does not attempt to execute Python code.

When run with Python, everything between triple quotes is regarded as a multiline string, so it is not parsed. Python does not execute any of the shell code inside the multiline string.

Code is valid in different programming languages is also called a polyglot. In this case, both pieces of code are quite readable, and not totally interwoven like most polyglots are.

Automatically setting up a virtualenv

The implementation details of the embedded installer should not through to the usage of the script in any way.

To be able to provide a seamless experience, we will need to:

1. Figure out where the script is located, so the virtualenv can be created in a folder next to it.
2. Automatically create the virtualenv and install dependencies.
3. Update dependencies when they have changed.
4. Kick off the Python script with the same environment, working directory and parameters.
5. Handling signals sent to the process

And finally, put everything together in one script.

Figuring out where the script is located

To be able to run our script from any directory, we have to figure out where the script is located. In a shellscript, parameters passed on the commandline are numbered $1 (first parameter) to $9 (9th parameter). The variable $0 is also available, and this variable refers to the script itself.

Let’s experiment a bit with the values that this variable can take in different situations.

1. The script is called directly with a full path. e.g.: lars@lars-debian:~$ /home/lars/my-tool/prog.py
2. The script is called with a partial path. e.g.: lars@lars-debian:~/my-tool$ ./prog.py
3. The script is on $PATH. e.g.: lars@lars-debian:~$ PATH="/home/lars/my-tool:$PATH" prog.py
4. The script is a symlink. e.g.: lars@lars-debian:~$ ln -s /home/lars/my-tool/prog.py bin/prog; ./bin/prog
5. The script is a symlink on $PATH. e.g.: lars@lars-debian:~$ ln -s /home/lars/my-tool/prog.py bin/prog; PATH="~/bin:$PATH" prog

Let’s create a very simple script, and place it in /home/lars/my-tool/prog.py. (Don’t forget to make it executable with chmod +x)

#!/bin/sh
echo "$0"

The different situations give following result:

1. Script is called directoy with a full path: $0=/home/lars/my-tool/prog.py
2. Script is called with a partial path: $0=./prog.py
3. The script is on $PATH: $0=/home/lars/my-tool/prog.py
4. The script is a symlink: $0=./bin/prog
5. The script is a symlink on $PATH: $0=/home/lars/bin/prog

All these situations need to be normalized to the canonical path, the actual location of the script.

The realpath command is used to resolve symlinks, and it also resolves relative paths.

With realpath, all the above situations result in the same path: $(realpath "$0")=/home/lars/my-tool/prog.py.

To cut off the filename of the full path, we can use the dirname command.

#!/bin/sh
SCRIPT_DIR="$(dirname "$(realpath "$0")")"

Automatically create virtualenv and install dependencies

The next step is to automatically create a virtualenv if none exists, and then immediately install the dependencies of the script.

The virtualenv is located in a hidden folder, .venv, that is placed in the same directory as the script itself. Dependencies are also in the same directory, in a standard requirements.txt file.

We assume that the venv python module is globally installed, because that is the preferred way to create a virtualenv in Python 3.

Because the script should stop immediately when any error occurs, we use /bin/sh -e as shebang. A failing shell command will cause the interpreter to exit immediately, instead of continuing to the next command.

#!/bin/sh -e
SCRIPT_DIR="$(dirname "$(realpath "$0")")"
if [ ! -e "$SCRIPT_DIR/.venv" ]; then # If .venv does not exist
    python3 -m venv "$SCRIPT_DIR/.venv" # Create a venv
    . "$SCRIPT_DIR/.venv/bin/activate" # Then activate it
    pip install -r "$SCRIPT_DIR/requirements.txt" # And install requirements with pip
fi

Update dependencies that have changed

The previous script installs dependencies correctly. However, it does not update the dependencies when a change is made to the requirements.txt file. You will need to remove the .venv folder manually and rerun the script to install new dependencies. Or, you need to activate the virtualenv manually and then run pip install -U -r requirements.txt.

We can improve this behavior by storing a hash of the requirements.txt file, and running pip again when the hash has changed.

#!/bin/sh -e
SCRIPT_DIR="$(dirname "$(realpath "$0")")"
if [ ! -e "$SCRIPT_DIR/.venv" ]; then # If .venv does not exist
    python3 -m venv "$SCRIPT_DIR/.venv" # Create a venv
fi
. "$SCRIPT_DIR/.venv/bin/activate" # Then activate it

REQUIREMENTS_HASH="$SCRIPT_DIR/.venv/requirements-hash"
# Create a requirements hash file when none exists yet
if [ ! -e  "$REQUIREMENTS_HASH" ]; then
    echo "0000000000000000000000000000000000000000  ../requirements.txt" > "$REQUIREMENTS_HASH"
fi
PREV_DIR="$(pwd)"
cd "$SCRIPT_DIR/.venv"
if ! sha1sum --check --status "$REQUIREMENTS_HASH"; then # If the hash does not match
    pip install -U -r "$SCRIPT_DIR/requirements.txt" # Install requirements with pip
    sha1sum ../requirements.txt > "$REQUIREMENTS_HASH" # Update requirements hash
fi
cd "$PREV_DIR"

Kicking off python with the same environment and arguments

We already learned about the $0 variable in a shellscript. It refers to the script file that is executed. The script parameters are in variables $1 to $9. But there is an other variable, $@, which is an array of all parameters that are passed to the program. "$@" can be used to get all parameters, without breaking parameters containing quoted spaces.

Before starting the Python script, it is necessary to activate the virtualenv first. When the virtualenv is activated, we want to run python with the same script file and parameters.

Since we are mixing Python and shell, we will start using the polyglot. We create a simple shellscript wrapper that directly calls a Python program. To verify that it is working, the Python program will print out its parameters and the working directory.

#!/bin/sh
''''set -e
python3 "$0" "$@"
exit
'''
import sys
import os
print(sys.argv)
print(os.getcwd())

We can see that the python code is executed, and the parameters are passed correctly.

lars@lars-debian:~/my-tool$ ./prog.py x -y --zz
['./prog.py', 'x', 'y', '-zz']
/home/lars/my-tool

The script also works correctly when using symlinks, or when called with a relative path.

Handling signals

There is a problem with the previous piece of code that may cause problems for programs that send signals to their children.

The direct child process will be the shell script instead of python.

lars@lars-debian:~/my-tool$ ./prog.py&; sleep 1; pstree -Aap $$
zsh,5878
  |-prog.py,21685 ./prog.py
  |   `-python3,21691 ./prog.py
  `-pstree,21698 -Aap 5878

Process 21685 is the shell script. This shell script has a child process, 21691, which is the Python program that is running.

Instead of running the Python program as a child process of the shellscript, we can replace the shellscript with the Python program. Sending signals from a parent process will now reach the Python process instead of being delivered to the shellscript, which would then need to propagate them to the Python process. It also looks nicer, because then there is no shell process sitting inbetween the caller and the Python program.

#!/bin/sh
''''set -e
exec python3 "$0" "$@"
'''
import sys
import os
import time
print(sys.argv)
print(os.getcwd())
time.sleep(200)

lars@lars-debian:~/my-tool$ ./prog.py&; sleep 1; pstree -Aap $$
zsh,5878
  |-pstree,22276 -Aap 5878
  `-python3,22268 ./prog.py

The exec shell builtin immediately replaces the shell process with a python process, so there is no need to add an exit afterwards. After all, the shell process ceases to exist after exec.

Putting everything together

Putting everything from the previous exploration together, we can create a script that automatically installs its PyPi dependencies from a requirements.txt file located in the same folder as the script.

#!/bin/sh
''''set -e
SCRIPT_DIR="$(dirname "$(realpath "$0")")"
if [ ! -e "$SCRIPT_DIR/.venv" ]; then # If .venv does not exist
    python3 -m venv "$SCRIPT_DIR/.venv" # Create a venv
fi
. "$SCRIPT_DIR/.venv/bin/activate" # Then activate it

REQUIREMENTS_HASH="$SCRIPT_DIR/.venv/requirements-hash"
# Create a requirements hash file when none exists yet
if [ ! -e  "$REQUIREMENTS_HASH" ]; then
    echo "0000000000000000000000000000000000000000  ../requirements.txt" > "$REQUIREMENTS_HASH"
fi
PREV_DIR="$(pwd)"
cd "$SCRIPT_DIR/.venv"
if ! sha1sum --check --status "$REQUIREMENTS_HASH"; then # If the hash does not match
    pip install -U -r "$SCRIPT_DIR/requirements.txt" # Install requirements with pip
    sha1sum ../requirements.txt > "$REQUIREMENTS_HASH" # Update requirements hash
fi
cd "$PREV_DIR"
exec python3 "$0" "$@"
'''
import sys
import os
print(sys.argv)
print(os.getcwd())
sys.exit(3)

I host a medium-sized Nextcloud installation that is quite actively used.

Recently, I was notified by my users that performance is extremely bad. Upon investigation, it was clear something bad was going on. Data transfer was multiplied by 40 compared to the previous period.

This increase in data transfer usage resulted in exhausting the allowed transfer per month after only 10 days. With the provider this server is hosted with, going over the allowed datatransfer does not result in extra charges. Instead, they throttle the bandwidth to 10Mbps.

The components

Nextcloud

The VPS also runs some less-bandwidth intensive services other than Nextcloud. For isolation and accounting purposes, a separate PHP-fpm pool is used for running Nextcloud.

To avoid it eating all RAM, the pool is limited to a maximum of 6 children. This means at most 6 requests can be handled in parallel. Under normal circumstances this limited amount of parallellism is not a problem. The maximum is hit occasionally, and in that case the new request has to wait until a previous request has finished.

When a file is being downloaded, this uses a PHP-fpm process for the full duration it takes to download the file. Because uploading a file is normally not limited by the networkspeed (the response is sent to cloudflare, not the end-user directly), this does not pose a problem and processes are freed up quickly again.

However, when bandwidth becomes the limiting factor, uploading the file becomes slower. This means the PHP-fpm processes will spend a longer time to serve a single request, quickly exhausting the available pool.

Here you can see that the active PHP-fpm processes almost never fall below the maximum anymore after the bandwidth cap is applied.

Key behavior: On all files served from the repository, cache control headers are set.

Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache

PDF.js

PDF.js is embedded in Nextcloud and used to display PDF documents from the filesystem.

Key behavior: Uses HTTP Byte-Range requests when the remote server supports them to fetch only the parts of the file that it needs.

Cloudflare

Cloudflare is put in front of the Nextcloud instance, acting as a barrier to block bad bots and to cache static files to reduce data usage and increase speed.

The cache settings of Cloudflare are set to its defaults, which means static files are cached and HTML is not cached.

Key behavior: Tries to serve Byte-Range requests from cache. If the file is not in the Cloudflare cache, it is requested in full to cache it. Cloudflare also respects Cache-Control headers.

The result: 20-30 times more data transfered than necessary

What happens when a user opens a PDF file?

1. PDF.js sends a byte-range request for the metadata of the PDF.
2. Cloudflare sends a request for the full PDF file. The request looks like a static file, so they probably want to cache it.
3. Nextcloud sends a no-cache header and sends back the full PDF.
4. Cloudflare receives the PDF, but can’t cache it. It responds to PDF.js with the partial content it requested and then discards the file.
5. PDF.js recognizes that the server supports byte-range requests, and keeps using them to load pages on demand.
6. User scrolls through the PDF. Every new page that needs to be loaded fires off an other byte-range request.
7. Requests are handled by Cloudflare, the full file is requested from the origin server and a partial response is sent back to the client.

Repeat this for multiple pages of a PDF when a user is scrolling through them. Most of the PDFs are scanned content, so they have reasonably large pages.

How to fix

You have 2 options if you want to fix this issue.

The first and most simple option is to simply disable Cloudflare for the domain you run Nextcloud from. Of course, this solution also has the downside that static assets of Nextcloud are no longer cached by Cloudflare and you no longer have the protection either.

The second option, which seems to work for now, is to put in a page rule in Cloudflare to bypass the cache for all Nextcloud WebDAV requests.

Results

After bypassing the cache for WebDAV requests, the amount of bytes transmitted through the Nextcloud WebDAV endpoint immediately dropped to a fraction of what it previously was.

A more detail view after applying the fix. Transfer rates went from 5 MB/s and over to mostly under 200 kB/s, which is a reduction of 25 times.

I'll try to describe the process I followed to create the parser here. It was a struggle to get to a minimal parser, but it certainly was an educational experience.

The parser is built in two pieces, a lexer and a compiler. This is something I remembered from how C compilers and friends are built.

The lexer

The lexer converts the string to tokens. It knows and validates the syntax of the 'language', and returns some stream of tokens representing keywords, strings, integers, ...

In my case, all tokens actually are strings, but they have a different meaning. The lexer derives the meaning from special characters in the neighborhood of the token. Usually the character is right in front, or right behind the token.

For example: title is just a string, but title: is a field name. So we associate the tokens T_STRING and T_FIELD_NAME with them.

You may want to have a look at the code while reading the next paragraphs, as my explanation makes a lot references to it.

To create the lexer, we loop over the string one character at a time. I prefer to do that with a while-loop instead of a for-loop, so I can choose whether to increment the loop variable or not.

Within the loop, we fetch a character of the string. Then a huge switch-statement executes the right code for the character we just read.

• '\': Escape character. Just append the next character as-is to the current token.
• <space>: the end of a token. Put it in the array and create a new one
• ':': If there is nothing in front (there is a space right in front): Syntax error. If there is something in front of it that has not yet a token id assigned, it is a T_FIELD_NAME, else syntax error. Put it in the array and create a new token T_FIELD_VALUE (that's what comes after a colon)
• '^': If there is nothing in front: Syntax error. If the thing in front has no token yet, it's T_FIELD_NAME, else it's an error. Put it in the array, and create a new token T_FIELD_WEIGHT. Read an integer into the token. Put it in the array, and if a colon follows, put the T_FIELD_NAME up again. (This simplifies the compiler a lot)
• '@': If there is something in front: Syntax error. Else, the current token is a T_SEARCH_FIELD.
• '"': Quoted string. It must have a space in front (or, the current token must not contain any text yet). Read an encapsulated string (up to the next quote).
• Anything else: Just append it to the current token.

All characters consumed? Return an array with the tokens for the compiler to use.

The compiler

The compiler converts the tokens it receives to actual code.

Basically, it's just another big switch-statement in a loop. But now, we loop over the tokens.

Using a foreach-loop is not possible, because we sometimes need to consume two tokens at a time.

So: If the token is a T_STRING: Append it to the search query.

If it's a T_FIELD_NAME: Take the next token, and make sure there is one.

• It's a T_FIELD_VALUE: call the addFilter() function with the field name and the field value.
• It's a T_FIELD_WEIGHT: call the addWeight() function with the field name and the field weight
• It's something else: Compile error: unexpected token

It's a T_FIELD_SEARCH: call addSearchField() with the field.

It's something else: Our lexer messed up. Compile error.

Conclusion

Simple parsers are quite easy to write manually, and are a lot faster than generated regex-based ones.

However, if it's the first parser you write, things can take a while figuring out. You'll have to bite the bullet, and start experimenting.

And don't forget to write tests! If you break a parser, and you don't figure it out immediately, you're gonna have a hard time fixing it.

The rings.lua script is not entirely mine. This guy created the base, I merely tweaked it.

Step 1: Get the stuff

Download my the Gist.
Install Conky through the command line, or use the software center.
$ sudo apt-get install conky
Note: the $ Just means you're typing this in a shell, you don't have to type it.

Move the files from the zip to their indicated place. You'll need to create a ~/.conky directory to move the scripts to.

Move the font files (.ttf) to /usr/share/fonts/.
$ sudo mv *.ttf /usr/share/fonts/

Step 2: Setting up

1. Set background.jpg as desktop background. (You should be able to figure this out yourself)
2. Make sure Conky starts up together with your session
3. Open Startup applications, click the Add button.
4. Type conky in the text box after Command.
5. Make the shell scripts executable $ chmod +x ~/.conky/*.sh
6. Log out and log in again to make sure Conky starts up.

Step 3: Tweaking

The .conkyrc is tweaked specifically for my computer, my number of cpus, and my screen size. It may not look nicely aligned on yours.

Fixing the alignment

1. Open up ~/.conkyrc in your favorite text editor. (The screenshots are made in vim, if you wonder ;))
2. On line 39, change the value of gap_x until the vertical borders of the conky window aligns with the border of the innermost rectangle on the background. Subtracting moves the screen to the right, adding moves the screen to the left.
3. If you can't get the screen to fit nicely horizontally, try adjusting border_inner_margin on line 32. (This may happen if you have a huge monitor which stretches the background)
4. On line 40, change the value of gap_y until the bottom border of the Conky window aligns with the border of the 2nd rectangle of the background. Substracting moves the screen to the bottom, adding moves the screen to the top. (I got this aligned by accident :))
5. Save the file. Conky automatically reloads the file, and restarts.

Fixing the number of CPUs

I happen to have 2 CPUs in my computer, but I left in the extension for 4 CPUs in comments.

1. Open up ~/.conkyrc, uncomment lines 76 & 77 (remove the #)
2. Remove lines 78 & 79, they add padding for the case of 2 CPUs.
3. Open up ~/.conky/rings.lua
4. Uncomment lines 100 to 127 (just remove --[[ on line 100, and ]] on line 127)
5. Update indexes on line 437, 439 and 441. Change the 5 to 7.
6. Save the file. Conky will automatically reload the file.

Removing battery configuration

Desktop users seem to have some problems with the battery configuration.

To remove the battery configuration, open up ~/.conkyrc, and remove line 98 and 99.
Next, Open ~/.conky/rings.lua and remove line 268 to 295.

If I missed something, or it is not working for you, please let me know in the comments. Positive feedback is also welcome, of course!

I took some time to configure my Ubuntu Gnome desktop. The background and the conky indicator are tweaked to match up perfectly in colors and borders. Want this configuration too? All files are available in a Gist
This is just a short post listing all my resources I used. I'll try to get full instructions up soon. Full installation instructions are available

Some extra gnome extensions I also use (on the screenshot left to right, top to bottom):

• Cpufreq
• Caffeine
• Lock keys
• Drop down terminal
• Dash to dock & Workspaces to dock
• Removable drive menu (not visible)
• Hi, jack! (not visible, only shows message area when mouse in right-bottom corner
• Alternatetab

For testing those libraries, I use Travis CI. Every commit I push to the GitHub repository gets tested by their CI (Continuous Integration) server.

Of course, I use Composer for managing dependencies on my PHP projects.

Usually everything works out fine, but sometimes the dependency installation fails because I reached the GitHub API limit. That's 60 calls per hour per IP address for unauthenticated requests. Since Travis runs a lot of tests for various projects, it is possible that an earlier project already exhausted the API limit.

Because composer keeps waiting for input of a username and password, Travis aborts the build after 10 minutes, and it shows up as errored.

The bad solution

I have seen other blogs suggesting to create an OAuth token, and just include it in your repository. That is a very bad idea! The token grants unlimited access to your account via the GitHub API. If someone else got a hold of it, they can do all kinds of bad stuff to your account.

The good solution

Travis supports encrypted environment variables. That is great news, as we can encrypt or OAuth token and put it in an environment variable, right? Now the only thing Composer needs to do is read the env variable and use that token to authenticate to the API.

Unfortunately, it is not that simple. Composer does look for an OAuth environment variable.

The ugly solution

Luckily, Composer does read ~/.composer/config.json, and merges it with the package's composer.json. So, it's just a matter of getting the environment variable in the configuration file.

Step 1: Encrypt the environment variable

Travis provides a gem that helps you to set up secure environment variables. Go ahead and install it.
Note: You'll need ruby and rubygems to install the travis gem.

[code language="bash"]gem install travis[/code]

Next, we need an OAuth token to reach the GitHub API. Go to the Applications section of your account settings, and add a personal API token.

It is time to encrypt the token. Navigate to the project directory, and encrypt the environment variable.

[code language="bash"]travis encrypt GH_OAUTH=4584c14558afe5580abec2d57c2256e6cb804cbf --add env.global[/code]

The encrypted variable gets added to the project's .travis.yml automatically.

Step 2: Write a script to generate ~/.composer/config.json

We still need a script to create the configuration file. It is a very basic script.

[code language="bash"]if [ "$TRAVIS_SECURE_ENV_VARS" = "true" ];
then
mkdir ~/.composer/
echo '{ "config": {"github-oauth":{"github.com": ' > ~/.composer/config.json
echo "\"$GH_OAUTH\"" >> ~/.composer/config.json
echo '}}}' >> ~/.composer/config.json
fi[/code]

First, it checks whether Travis has secure env variables enabled. For security purposes, they get disabled when testing a pull request.

Then, create the ~/.composer directory and write the configuration to the configuration file.

Step 3: Add setup script to .travis.yml

Finally, the script has to be run before any Composer command is executed. Add the setup script in .travis.yml. You may also need to chmod +x the script before it will execute.
To disable all interactive questions Composer may ask, add COMPOSER_NO_INTERACTION=1 to your environment variables.

That's it

You may want to do this for every project using Composer and Travis. It is important to run travis encrypt separately for each project, as the encryption key is different for each project.

If you rather learn by example, you may want to look at vierbergenlars/defer@8eae9b

Update: The nice guys from travis-ci have added an API token for GitHub.