Techies 4 Temple Street 2016

Temple Street Children’s Hospital is Ireland’s leading paediatric hospital, caring for more than 145,000 children every year, and so is a charity which is important to many of us here at Swrve.

Techies4TempleStreet is an annual treasure hunt aimed at raising funds for the hospital. In 2015, over 800 attendees (including two teams from Swrve) managed to raise over €150,000. This year the event was set to be even bigger!

Each team pledges to raise €500 for Temple Street, so we hosted a bake sale, which contained all manner of delicious homemade and bakery-bought treats.

image04.jpg

Thanks to support from our fellow Swrvers as well as the Financial Services Union and Analytic Partners who we shared a building with, the bake sale was a huge success!

image08.jpg

Everyone was very excited to set off on a day of adventure! Our volunteers were split into two teams, Team Aces and Team Supremos.

When our teams arrived at the starting venue, they were given a map with sixty-four points of interest all over Dublin city centre. Each number marked on the map had a corresponding question, which you could only figure out by visiting these locations;

image05.jpg

Without a photographic memory and an extremely varied knowledge of every single square meter of Dublin itself, our teams had quite a bit of ground to cover in very little time!

But thankfully Adam H had an ingenious plan….

image01.jpgRegister, run around, solve clues, win. Easy!

Perhaps we should have broken these tasks down into smaller story points… but the registration deadline was upon us!

image02.jpgTeam Aces registered and ready to go!

Along with the map and corresponding questions, there were a number of mini challenges for our teams to complete. For example there was the hunt for Periscope Pete, who gave mysterious clues about his whereabouts over twitter during the event.

image00.jpgPeriscope Pete, caught by Team Supremo!

There were also an assortment of challenges posted to twitter during the day which needed picture proof, like hugging trees.

image07.jpg

Or getting selfies with the Gardai!

image09.jpg

After three long hours of adventuring, during which they covered over 16km of ground, our teams raced back to the RDS to hand in their completed answers, and take a much needed rest.

image03.jpgA very tired Bob 😦

With good reason to be exhausted, because our teams came third and seventh!

And even better;

We raised €1,135.40 this year, with the overall total raised by the whole event coming to an enormous €200,000!

image06.jpg

A celebration of beer and pizza was much deserved, and perked our hard working teams back up after a long day.

If you want to see more of how the day went, Ciaran has a fantastic video montage of the whole event;

Cycle Against Suicide 2016

At Swrve we are given two Volunteer Days every year, with which we can take part in charity or community activity as additional paid time off.

We live in a broader community where things are frequently not so rosy. Swrve would like to make a tiny dent in that by supporting any team member who wants to contribute work time to volunteer community activity.

– Hugh Reynolds, Co-Founder

In 2015 two of our team here, Ciaran and Tim took part in Cycle Against Suicide.

Ciaran on his way during Cycle Against Suicide 2015
Ciaran on his way during Cycle Against Suicide 2015

Over 800 people die by suicide in Ireland every year, and globally the rates of suicide have increased 60% in the past 45 years. Cycle Against Suicide aims to get people talking about mental health, and break the stigma associated with asking for help.

Mental health is an issue that affects everybody at some point during their lives; either directly or by supporting a friend or family member through a hard time.

For 2016, a few kind Swrvers donated their own Volunteer Days to allow Ciaran to do the full 2 weeks, joined by Tim and Dom.

Ciaran, Dom and Tim taking a much deserved break during Cycle Against Suicide 2016
Ciaran, Dom and Tim taking a much deserved break during Cycle Against Suicide 2016

In Ciaran’s own words;

It’s an unbelievable experience, something I can’t put words to. It’s a force of change and an amazing community of people.

He also filmed his journey of around 1400 kilometers!

Congratulations to all involved in the cycle this year, and hopefully the 2017 event will be even bigger! As the Cycle Against Suicide mission statement says;

It’s OK not to feel OK and it’s absolutely OK to ask for help.

Standing desks at Swrve

Here at Swrve we’ve got a world class engineering team. We’re an active lot. It turns out quite a few of us get restless when we sit at our desks for too long. Whether it’s cycling 1400km  around Ireland, hiking in the hills at the weekend, hitting the crossfit gym in the morning on the way to the office, swimming in the sea, taking part in running races, or just cycling to work a few days a week, it’s safe to say sitting stationary at a desk wasn’t what drew many members of the Swrve engineering team to our profession. Mix this up with the usual lower back pain most of us have suffered at some point in our working lives and it was inevitable that we’d end up trying out standing desks in our office at some point.

Back in May one person used their Friday Skunkworks time to build a standing desk from an Ikea coffee table. Quite a few members of the engineering team took the opportunity to use the new desk for a few days to evaluate whether it was a setup that they liked. We also picked up a commercial desk from Varidesk at around the same time which has been used by team members following back problems.

So far the feedback has generally been positive with three other standing desks appearing in the intervening months since May, including a second purpose built model from Ikea.

IMG_20150908_095819624

Fortunately we’ve been unable to reproduce the results reported in The Onion about how standing at a desk can increase coworkers disdain 😉

A few comments from the team on their standing desk experience:

  • “I found it pretty good, but tiring without an adjustable desk or a stool to sit on occasionally — standing all day was too much.”
  • “I used the pomodoro method to build up my endurance with the standing desk: first day 25 mins standing, 5 mins sitting, second day 30 mins standing, 5 mins sitting”
  • “A standing desk is really great for being productive and focused on the task, but if I have to do a lot of thinking about a problem, I find myself sitting down”

We’re looking forward to getting our hands on a copy of Kelly Starett’s Deskbound when it becomes available.

Announcing RateLimitedLogger

Logging is vital for production-ready, operable code; however, in certain situations, it can be dangerous. It is easy to wipe out throughput of a performance-critical backend component by several orders of magnitude with disk I/O in a performance hotspot, caused by a misplaced log call, or input data changing to something slightly unexpected.

RateLimitedLogger is an Apache-licensed, SLF4J-compatible, simple, fluent API for rate-limited logging in Java. A RateLimitedLog object tracks the rate of log message emission, imposes an internal rate limit, and will efficiently suppress logging if this is exceeded. When a log is suppressed, at the end of the limit period, another log message is output indicating how many log lines were suppressed.

This style of rate limiting is the same as the one used by UNIX syslog; this means it should be comprehensible, easy to predict, and familiar to many users, unlike more complex adaptive rate limits.

Once the rate limit is hit, logging becomes just a couple of java.util.concurrent.AtomicLong operations, so it’s very inexpensive, performance-wise.

More details are in the README.

How we increased our EC2 event throughput by 50%, for free

(TL;DR: using an SSD cache in front of EBS can give a massive I/O throughput boost)

Background

Internally, Swrve is built around an event-processing pipeline, processing data sent from 100 million devices around the world each month, in real time, with an average events-per-second throughput in the tens of thousands.

Each event processor (or EP) stores its aggregated, per-user state as BDB-JE files on disk, and we use EBS volumes to store these files.  We have a relatively large number of beefy machines in EC2 which perform this task, so we’re quite price-sensitive where these are concerned.

We gain several advantages from using EBS:

  • easy snapshotting: EBS supports this nicely.
  • easy resizing of volumes: it’s pretty much impossible to run out of disk space with a well-monitored EBS setup; just snapshot and resize.
  • reliability in the face of host failure: just detach the volumes and reattach elsewhere.

EBS hasn’t had a blemish-free reliability record, but this doesn’t pose a problem for us — we store our primary backups off-EBS, as compressed files on S3, so we are reasonably resilient to any data corruption risks with EBS, and in another major EBS outage could fire up replacement ephemeral-storage-based hosts using these.  In the worst case scenario, we can even wipe out the BDB and regenerate it from scratch, if required, since we keep the raw input files as an ultimate “source of truth”.

One potential downside of EBS is variable performance; this can be addressed by using Provisioned IOPS on the EBS volumes, but this is relatively expensive.

However, our architecture allows EPs to slow down without major impact elsewhere, and in extreme circumstances will safely fall back to a slower, non-realtime transmission system.  If things get that slow, they generally recover quickly enough, but in the worst-case scenario our ops team are alerted and can choose to reprovision on another host, or split up the EP’s load, as appropriate.  This allows us to safely use the (cheaper) variable-performance EBS volumes instead of PIOPS; it turns out these can actually perform very well, albeit in a “spiky” manner, with occasional less-performant spikes of slowness.

As we were looking at new EC2 instances several months back, we noticed that decent spec, high-RAM, high-CPU instances were starting to appear with SSDs attached in the form of the “c3” instance class.  These SSDs, in the form of two 40GB devices per instance, were far too small to fit our BDB files with enough headroom for safety, unfortunately.  But could they be used as a cache?

Our data model tends to see lots of accesses to recent data, with a very long tail of accesses to anything older than a few days, so there was a good chance caching would have a good effect on performance.  Let’s find out!

Continue reading