Cloud Networking and Monitoring with VyOs

Some useful hints and tips from my experience working with VyOs (http://vyos.net) routers - a fork of the, now Brocade-owned, Vyatta linux-based router OS - in AWS, KVM and Xen environments.

IPSec, BGP, AWS and VTI - tunnels dropping

I had (and still suffer with) an issue where periodically tunnels with AWS will drop, the ipsec and ike security associations will re-negotiate and establish a tunnel but the vti will not get brought back up.

This manifested as us seeing bgp peerings going offline and traffic no longer passing through some routers.

I've not been able to look in depth at the cause of the problem in VyOs yet - but if time ever becomes available I will do - but I use the following script as a hacky workaround, triggered from cron (specified using the "set system scheduled-task" commands) every 15 minutes, we should be quite well covered with this since we have a minimum of two routers connected to each VPC, each with two tunnels to the AWS VGW so we would have to lose 4 tunnels to see any actual outage:

#!/bin/vbash

down_ifs=$(/opt/vyatta/bin/vyatta-op-cmd-wrapper show interfaces | awk '$3 ~ "A/D" {print $1}')
failed=0

for iface in $down_ifs
do
  vpnpeer=$(/opt/vyatta/bin/vyatta-op-cmd-wrapper show interfaces vti $iface | awk '$3 ~ "peer" {print $NF}')
  vpnlocal=$(/opt/vyatta/bin/vyatta-op-cmd-wrapper show interfaces vti $iface brief | awk -v i=$iface '$1 ~ i {print $2}' | awk -F/ '{print $1}')
  bgpneigh=$(/opt/vyatta/bin/vyatta-op-cmd-wrapper show ip bgp neighbors | egrep -A 1 "^Local.*${vpnlocal}" | tail -n1 | awk '{print $3}' | sed 's/,//')
  ipsecstate=$(/opt/vyatta/bin/vyatta-op-cmd-wrapper show vpn ipsec sa peer $vpnpeer tunnel vti | awk '$1 ~ "vti" {print $2}')
  ikestate=$(/opt/vyatta/bin/vyatta-op-cmd-wrapper show vpn ike sa peer $vpnpeer | egrep 'up|down' | awk '{print $1}')

  if test "x$ipsecstate" == "xup" -a "x$ikestate" == "xup"
  then
    sudo /sbin/ip link set $iface up
  else
    echo "Interface $iface down, ike state $ikestate, tunnel state $ipsecstate, bgp neighbour $bgpneigh"
    failed=1
  fi
done

exit $failed

Monitoring in the cloud with DataDog

We monitor our cloud systems (and a number of on-premise systems as well) with DataDog, which has agents for all major Linux distributions as well as Windows and a few other unixes.  This is great - VyOs being debian-based it should just work, right? uh-oh - no dice.

VyOs handles updates and config rollbacks in a similar way to Juniper - its a neat solution, although the rollback can be a but cumbersome - however, it does so by mounting the filesystem with overlayfs.

This causes problems for datadog as unix sockets created on overlayfs are buggy at best, and cause problems for many different systems.  This affects the DataDog agent, and my solution was to use ansible to make the following changes to the agent configuration and init scripts as a work-around, given on VyOs the /tmp folder is mounted as tmpfs:

Replace in /etc/dd-agent/supervisor.conf:

sed -i 's!/opt/datadog-agent/run/datadog-supervisor.sock!/tmp/datadog-supervisor.sock!g' /etc/dd-agent/supervisor.conf

Replace in /etc/init.d/datadog-agent:

sed -i 's!^SUPERVISOR_SOCK.*$!SUPERVISOR_SOCK=/tmp/datadog-supervisor.sock!' /etc/init.d/datadog-agent

and thats it - you can configure the agent and start the process as normal now - watch updates to dd-agent and version updates of vyos as you will likely have to re-apply those changes.

oVirt Testing - English instructions for raising swap threshold

oVirt 3.2 Small Test System Admin Notes

I'm testing oVirt in the office to get a few things straight - I'm going to try and keep track of my experiences here.

For comparison, I'm running oVirt 3.2 on Fedora 18.

Can't start VM - Host swap percentage error

My first problem (I'm using a fairly constrained single-host setup initially) is that I am very short of memory on my test host.  oVirt refuses to start VMs when swap usage is over a threshold.  I couldn't find anywhere in the management pages to set this threshold, and a good bit of googling returned mainly development threads on the subject of memory and swap handling in larger environments.

Below I have shown the solution to the error message I received:

Cannot run VM. Host swap percentage is above the defined threshold. – Check your configuration parameters for Host Swap Percentage.

 To increase the defined threshold (or just check the current value), here are the commands:

* Check the current value:

[root@ov01 ~]# engine-config -g BlockMigrationOnSwapUsagePercentage

* Expected output:

BlockMigrationOnSwapUsagePercentage: 0 version: general

 * Set the value (to 75% usage in this example):

engine-config -s BlockMigrationOnSwapUsagePercentage=75 -cver=general

* Re-start the engine service:

systemctl restart ovirt-engine.service

Hope this helps someone, I'm sure I'll come back to it again if I build out any more small test systems

Earrings at School PE

At the beginning of the summer holidays, my 8 year-old step-daughter had he ears pierced, as did a number of her classmates.

Unfortunately, upon her return to school, her piercings have not healed enough for her to be able to remove the studs.  Since our Local Education Authority has a policy of absolutely no jewellery in PE lessons, and has said she will not be able to take part in PE without removing her earrings, I contacted the Association for Physical Eduction (http://www.afpe.org.uk) by twitter (https://twitter.com/jimrippon/status/245907146166906880) to ask what their advice would be so that she wouldn't miss out on her lessons.

Within a few minutes I had an excellent reply from Simon Leach, Membership & Communications Officer.  I have included the response below to help any parents who are confused or concerned, and make sure we are all approaching the schools we work together with to provide our children the best education we can.  I hope this helps.

Hi Jim,

Here’s some info about earrings in PE.

Kind regards,

Simon Leach
Membership & Communications Officer
Association for Physical Education

Mobile:
Tel:
Email:
Web: www.afpe.org.uk

The attached memo follows:

afPE Health and Safety Update – January 2011

Earrings

Well this is my first offering for the H & S update, having taken over the Safety Officer role from Glen as from the start of this year.  I have already had a wide variety of enquiries through the helpline, and have chosen to recap here on a familiar issue which can cause headaches for primary and secondary alike, earrings and PE.

The enquiries on this subject have come from irate parents who want to clarify what the correct line is,  and so schools need to be sure they are clear, and that their PE policies, and/or school prospectus outline exactly what they intend.

Whilst it would be ideal if ears were pierced at the start of the summer holidays, given six weeks to settle and are then ready to remove for PE lessons as from September, this is frequently not achieved.    Where a Local Authority has a “no jewellery” statement in their policy, schools must follow that policy.  For those authorities who have adopted afPE’s Safe Practice guidance as the basis for their H & S policy for PE, this advocates that earrings are removed before a PE lesson. Where this cannot happen one alternative is that the earrings can be covered with tape by the pupil, or the parent before coming to school.  However it is still the responsibility of the member of staff leading the lesson to ensure that the tape is doing the job intended, and then to assess if this is sufficient precaution to allow the pupil to take part, either fully or partially in the planned lesson.

Where neither the earrings can be removed, or taped over, the member of staff still has a legal duty to consider how the pupil can participate in the lesson in some way. This process of assessment could involve considering amending the tasking, using different equipment etc. although having gone through this process the decision may still be that safe active participation is not possible.  Alternatives at this point could include allowing the pupil to be part of the planning aspect, in a gymnastics sequence, discussing and evaluating tactics within a game playing activity, or taking on an officiating role. (Safe Practice, 2008 pg 100 -14.2.2.)

What has got the parent’s collective goat in recent enquiries I have received has been:

•  A school who ‘changed’ their policy but failed to update their website with this amendment.
•  A school who totally excluded two sisters from PE because they could not remove the earrings.
•  A lack of willingness to come to some agreed action to deal with the situation.

In reality this should be a straightforward issue to deal with; set policy, clear communication, flexibility to ensure learning continues to take place, are really the three things that need to be considered.

Angela James

afPE Safety Officer

 

Open Source and Security

Just a quick brief note - as again I have been riled by someone claiming that closed source is inherently more secure, because with open source an attacker "knows more about the underlying architecture"

In simplest terms, being able to knit does not make counterfeiting a particular jumper any easier - knowing the source code of a product should not provide an attacker any more capabilities than a similarly secure (or otherwise) closed source product.

In fact, I believe a fair amount of complacency comes with developing closed source code leading to more difficulty in creating inter-operable applications and the potential for a greater number of security vulnerabilities.

Ok, rant over...

Google Horoscope makes me chuckle

My iGoogle horroscope today made me smile - right on the money:

Although you may have obligations that carry you out into the world today, you may be more willing to get involved if you are assured that you'll have time to yourself later on. You have been so busy recently that you need some downtime in order to sort out your feelings and to tune in with your intuition. Remember, being with other people is not always a distraction; it can be part of your process, too.

Had a very busy few weeks and months, and having been ill most of this past week, I can definitely do with some down time this weekend - I'll be home with the kids tonight so reckon a nice bottle of New Zealand Marlborough Sauvignon Blanc and something nice to eat will do the trick. Just need to find something to keep them occupied for the day - looks like it'll be a trip to the seaside for the afternoon.