In this post We detail how to use the VPN Service within openstack liberty using UKFast ECloud Flex.
To start you will requrie a login to the horizion dashboard and navigate to the VPN Section of the dashboard.
We start by setting up a IKE Policie - The config I used in this example is below:
Name: MyFirstVPN Description: MyFirstVPNDescription Authorization algorithm: sha1 Encryption algorithm: aes-128 IKE version: v2 Lifetime units for IKE keys: seconds Lifetime value for IKE Secrecy: 1200 Perfect Forward Secercy: group14 IKE Phase1 negotiation mode: main
We then Create an IPSec Policy - The config I used in this example is below:
Name: MyFirstVPNIPSecPolicy Description: MyFirstVPNIPSecPolicyDescription Authorization algorithm: sha1 Encapsulation mode: tunnel Encryption algorithm: aes-128 Lifetime units: seconds Lifetime value for IKE keys: 1200 Perfect Forward Secercy: group14 Transform Protocol: esp
We then Create a VPN Service - The config I used in this example is below:
Name: MyFirstVPNService Description: MyFirstVPNServiceDescription Router: 1986 (This should be the router you want the VPN to run within) Subnet: 10.0.0.0/8 (This should be the subnet you want the VPN to run within) Admin State: up
Finally We Create an IPSec Site Connection - The config I used in this example is below:
Name: MyFirstVPNService Description: MyFirstVPNServiceDescription VPN Service associated with this connection: MyFirstVPNService IKE Policy associated with this connection: MyFirstVPNService IPSec Policy associated with this connection: MyFirstVPNIPSecPolicy Peer gateway public IPv4/IPv6 Address or FQDN: 1.2.3.4 Peer router identity for authentication (Peer ID): 1.2.3.4 Pre-Shared Key (PSK) string: MyReallySecurePSKHASH
Now our VPN Service is setup and we can configure the remote side to connect - The below is taken from StrongSwan running on a remote linux server, This should apply similarly to Openswan. Other VPN gateway software may require different configeration
/etc/ipsec.conf conn %default ikelifetime=60m keylife=20m rekeymargin=3m keyingtries=1 keyexchange=ikev2 conn openstack left=1.2.3.4 leftsubnet=172.31.255.0/24 leftid=1.2.3.4 right=$REMOVED rightsubnet=10.0.0.0/8 rightid=$REMOVED auto=add authby=secret /etc/ipsec.secrets]]>: PSK "MyReallySecurePSKHASH"
Having a point in time backup of your critical Mysql Data can be a life saver when ‘something bad happens’. This post explains how we can use the power of innobackupx, quicklz, xbstream and s3cmd, to take a none locking hot dump of your Mysql Data, Compress it using quicklz, encrypt it using AES256 and stream it into Ecloud Vault with the help of XBSteam and s3cmd.
The below outlines the Software Requirments you will need to walk with this guide.
1 2 3 |
|
You can Find innobackupx and qpress iin the Percona repo here You can find s3cmd in the epel repo here
You will need to ensure you can login to mysql without a password at the command line, this can be done by configuring a ~/.my.cnf file - an example is below
1 2 3 |
|
You will them need a copy of the latests backup scrip you can find that on my github here
I place this file under /usr/local/bin and make it executable - the below should do this for you
1 2 |
|
Now you can edit the backup script to modify the below variables at the top of the file:
1 2 3 4 5 6 |
|
You can find your Vault access details here, You should create a new bucket for each server you backup - You can create a bucket here or using the below s3cmd options
1
|
|
The first time you run the script you will need to setup an encryption key to ensure the data is stored securely. We use AES256 for this as it is currently very difficult for even a state sponsored attack to break - You Must keep a copy of your encryption key otherwise you won’t be able to restore your data later. The below is used to setup the key for the first time.
1 2 3 4 |
|
I can stress enough how important this key is for when you come to restore your data, The password you entered will not be enough to regenerate the key.
Next you can run the backup for the first time using the below:
1
|
|
You should watch the output to ensure you see “completed OK!” at the end of the output.
You may then wish to run the backup daily by adding the below to cron.daily
1
|
|
If you wish to restore a backup you will need to download the backup file from vault using s3cmd and follow the below, I assume your backup is called today.full.xbstream, that ./restore/ as enough free disk space to extract the backup and your AES-256 encryption key is located in /root/.vault.key - you may need to edit these details for your case.
We need to extract the xbstream file
1
|
|
We then need to remove the AES encryption using xbcrypt
1 2 |
|
We then need to remove the quicklq compression using qpress
1
|
|
We then need to apply the log file
1
|
|
You are then ready to copy the mysql data dir back into your mysql server
1
|
|
I run a PB scale ceph cluster providing RBD and RGW access, We are used to ceph’s point release updates and the advice from Sage to update existing clusters to the new version as soon as possible.
I had lot of experience upgrading ceph clusters all the way back from Hammer to current Firefly, The process has always been to update the mon’s followed by the OSD’s and finally the clients in my case RGW’s and Openstack Clients. But the process of doing updates manually was time consuming and very dull - To the rescue comes ansible, this the below playbook we are able to upgrade a PB scale cluster from 0.94.5 to 0.94.6 in just over 1hr fully unattended.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 |
|
I ran into a case where a popular website was receiving a large number of undesirable http requests (>20k per second) that the application was unable to reject efficiently with a 400 range status code without adding more web servers.
It was quickly apparent the traffic all originated from single user-Agent that for the purpose of this post we will call ‘BADAGENT’
The first attempt to block this traffic was done using varnish, this worked well but required multiple servers with high end CPU’s to analysis the http headers and drop them - The code used for this is below
1 2 3 |
|
The below was also used to try and block based upon the URL not just the User-Agent to test how flexible this was:
1 2 3 |
|
After reading good reviews about haproxy Performance, I moved to dropping traffic in haproxy and then sending it back into varnish - This dramatically reduced the CPU usage compared to varnish but did have the undesirable effect of having to proxy traffic to a high end port number when sending traffic back into varnish - The code used for this is below
1 2 3 4 5 |
|
I also found it possible to provide a file containing a line separated list of BADAGENTS as follows
1 2 3 4 5 |
|
It was also possible to use the above to block on url parameters such as http://domain.com/my_api.php?var1=userbob - We could use the below to block on the url key value pair of ‘userbob’
1 2 3 4 5 |
|
This really stabilised the environment but I really wanted rid of haproxy so using the Solarflare Communications SFC9120 card with the Filter Engine licence It was possible to drop the traffic in the network card before it even hit Linux - This not only saved CPU but also saved bandwith in responding to the http requests. The bandwidth saving was to the tune of > 50TB PCM.
The code used to do this on the card for a User-Agent is below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
|
It is also possible to filter on items within the URL with the below:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
|
This was the ideally solution - there are many more possible options with the SolarFlare card, full details with more examples can be found in the user guide
]]>Supporting a user today we hit a bug that provents the user from uploading volume to image using the dahsboard. As always with the dashboard the error is not helpful but we know this will be fixed int he ‘M’ release cycle.
Using the command line as follows we got a much better error message:
1
|
|
The error:
1
|
|
We tracked this error back to LP Bug 1527324, And found a fix had been commited into openstack/cinder 8.0.0.0b2 development milestone, But this did not help us in production today.
So the work around, We first had to list the image to via the meta data:
1
|
|
Then we can loop over the meta data items and remove them:
1
|
|
This then enabled the image to be created from the volume
]]>Oftern in ceph you may want to change the default Disk schedular to ensure you can deliver the best expereience for your users. For my usecase we needed to ensure that client I/O takes prioirty over Recovery So I set the below options in ceph.conf on the osd servers.
1 2 |
|
As the above onyl work with CFQ and the default schedular was deadline I added the below to the grub config to change this.
1
|
|
However as I use SSD’s for the Journals I wanted to ensure these disks use NOOP so I used the below udev rule for this:
1 2 |
|
In this post we look at how to take a VirtualBox image (ova) file and use it in openstack backed by ceph. In this example I will use an image provided by kali.org.
To start you will need a Linux instance with around 4 CPU’s and 4GB or RAM, less will work but I was on a time limit.
Next we download the image using wget
1 2 3 4 5 6 7 8 9 10 11 |
|
Then we extract the image
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
|
Now we can extract the ova file to review the VMDK files inside:
1 2 3 4 5 6 7 8 9 10 11 |
|
Now we convert the VMDK to a raw file - a qcow2 would also work here but as my openstack install is backed by ceph I am using RAW
1
|
|
Finally we upload the converted image to glance using the glance phyton-client
1
|
|
Now are image is ready to be used and we can boot an instance from it
]]>When using qemu-KVM in openstack to support a Windows Guest I noticed On highload hypervisor tap interface of net-highload guest drops many TX packets. The below shows an example of what this looks like from the hypervisor.
1 2 3 4 5 6 7 8 |
|
This was causing the guest to have high jitter and stability problems, after some investigation it was clear that increasing the the txqueuelen on the tap interface resolved the dropped packets problem and the jitter and stability problems in the guest. This was then added to udev using the below so that new created interfaces and existing interfaces would get an increased txqueuelen.
1 2 3 4 5 6 7 8 9 10 11 |
|
Just a short into I created for UKfast’s ecloud Flex - but it a general walk through the Horizion Dashboard {Kilo Release}
]]>First We locate the Server UUID and network UUID
1 2 3 4 5 6 |
|
1 2 3 4 5 6 |
|
Now using the above We add an extra fixed IP
1
|
|
Then we check the IP is added
1 2 3 4 5 6 |
|
Now you can add the extra IP within the instance’s Operating system - This will very depending upon the image you have used, linux, windows BSD etc…
]]>First we need to install the openstack python clients, You will need a Linux instance with root access for this - I used ubuntu 14.04
1
|
|
Now we need to create a credentials / RC file - This will pre set a number of environment variables for later use - Fill in the
1 2 3 4 5 |
|
Now we just source the credentials / RC file - The below assumes you called the file my.rc
1
|
|
Now we can start to make use of the python packages we have installed - the below are a few examples to get you started, but you should run the commands with no arguments to view the help pages and explore the possibilities.
1 2 3 4 5 6 |
|
1 2 3 4 5 6 7 8 9 10 11 |
|
1 2 3 4 5 6 |
|