Wednesday, September 30, 2015

Automation for DBA - Ansible part 1

Hello,

In this post I would like to move forward with software provisioning and configuration. In my last post I covered a ways to build a "core VM" and now it's a time to add some software and configuration to it.

There are couple of automation tools, which can be used for that task like Puppet, Chef or Ansible to name a few. The latter one - Ansible - is my favorite, cause in my opinion it has shortest learning curve and also doesn't require any agents on the remote servers.
Although there are some requirements which has to be met on the remote servers:
- ssh connectivity to the remote host
- Python 2.4 ( with python-simplejson ) or 2.5 installed on the remote host.

If Python is a "big no-no" in your organization, you can still you Ansible to help you run a script over set of servers but you will loose almost all functionality.

How Ansible works:

Ansible has a concept of the control machine and a remote managed servers. Ansible scripts are started on the control machine and other servers are managed over a SSH connection and Ansible modules send to the remote servers on demand. Ansible can run in the fully automated mode only if control machine and remote servers has a password-less SSH configuration. In the other case Ansible can ask for a SSH password.

The control machine can be a dedicated sever or it can be an OS admin or DBA laptop, where Ansible is installed. Please check Ansible documentation to find a information how to install it. 

Inventory
Remote servers can be managed from control machine, only if they are added into a Ansible inventory. The inventory is a text file with the following format:
testdb.mycompany.com

[testgroup]
envtest ansible_ssh_host=172.16.180.190 ansible_ssh_user=oracle

[proddb-group]
rac1.mycompany.com
rac2.mycompany.com

There are two groups in inventory file - testdbgroup with one members envtest, and proddb-group with two members rac1 and rac2. Server - testdb - is listed in the inventory file but it not a part of any group. 

Variables
Ansible has a concept of host and group variables. 
Group variables are used for all host defined in the a group. There is a directory called group_vars, containing a files named with a group name. Each file is a set of variables for a particular group.
Host variables are used for a particular host only and they overwrite a group variables. There is a directory called host_vars, containing a files named with a host name. Each file is a set of variables for a particular host.

$ ls -l host_vars/
total 24
-rw-r--r--  1 mprzepiorowski  staff  29 Sep 29 12:44 envtest

$ cat host_vars/envtest
password: "host env password"

$ ls -l group_vars/
total 8
-rw-r--r--  1 mprzepiorowski  staff  23 Sep 29 13:10 testgroup

$ cat group_vars/testgroup
groupname: "Test group"
 
Server envtest has a file with host variables and it belongs to the group testgroup with a defined variables as well. Both will be used in the following example. 

Running mode
There are three ways of running a remote command:
- single command mode, 
- Ansible playbooks
- Ansible playbooks with roles
and commands or playbooks can be run on the specific host from inventory file, specific group or all hosts.

Examples:

Single command:

In the first run Ansible can be used to add SSH public key from control machine into remote server. By default Ansible connect to the remote servers using a user running a command on the control machine, but this can be changed in the inventory file.
$ cat inventory/lab
[testgroup]
envtest ansible_ssh_host=172.16.180.190 ansible_ssh_user=oracle
Remote server has a user oracle and it will be used by all scripts. If remote server name "envtest" is not resolved by a DNS, IP address can be added to inventory file as well.

Test a SSH connection to remote host
$ ssh oracle@172.16.180.190
oracle@172.16.180.190's password:
Password is required and this is an example command to fix this problem

$ ansible -k -i inventory/lab all -m authorized_key -a "user=oracle key=\"{{ lookup('file', '/Users/mprzepiorowski/.ssh/id_rsa.pub') }}\" "
SSH password:

envtest | success >> {
    "changed": true,
    "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD43SwP3zXR/LrKqADJDC8jUOv0vgMEQVqWOXoUgwlzQ1vpS1I5m3GjXggG7fjU5I2jGbG+V9xSqPZdw4MGed4nsgOo1m68w9oBpfYLu3hKACSw1Tgu8Ghxd5TB9ohyZ7NOgepXB+zmV8NoqvAUg4yDkk4qPn1fQmoIz7yLkPRErnmSkI6e/gHsGAmZ+5WdvH0FByPCMEfHqDe8vI4ZPVbPWZzl5x9m72HtxjCbB2htHg1JEcch6927oiRE9rllbOL0M/tw5LgjCtYpM3iZTsBwFsGZ6NQ1DM+OiJSrDEhUqUMOZ08X/G7aNYb28QUt9G/FZTTwZjEfnOO75n3i29U/ mprzepiorowski@Marcins-MacBook-Pro.local",
    "key_options": null,
    "keyfile": "/home/oracle/.ssh/authorized_keys",
    "manage_dir": true,
    "path": null,
    "state": "present",
    "unique": false,
    "user": "oracle"
}
Command options:
- k - ask for SSH password
- i inventory/lab - use a inventory file from folder inventory and file name called lab
all - run this command for all hosts from inventory file
-m authorized_key - use Ansible module authorized_key
- a "user=oracle, key=xxxxxxxx" - Ansible module parameters 

Command output:
- changed : true - remote system has been changed due to Ansible call
- envtest | success - action returned OK status
- other lines are depend on the module


Ansible returned OK status so now SSH connection should work without password:
$ ssh oracle@172.16.180.190
Last login: Mon Sep 28 15:58:24 2015 from 172.16.180.1
[oracle@envtest ~]$
Ansible did a work and SSH public key has been added to user configuration on the remote server. 

Single command is a nice feature if you want to run single command across fleet of servers, but for most of cases Ansible playbook will be a way to deal with.

Ansible playbook:

Next example will show how to create a simple playbook with an one task and install a vim package using yum

Playbook file is formatted in YAML and it looks like this:
---
- hosts: all
  tasks:
     - name: install vim
       sudo: yes
       yum: name=vim state=present
Be aware that YAML file syntax include a white space, so format of this file matter as well.

Line description:
Hosts: all - mean run a playbook for all hosts from inventory file
tasks: - is a start of task lists (one task in this example)
name: - start of task definition
sudo: - run this module using sudo
yum: - module name with parameters (name is a package to install, state=present mean install it)

Ansible playbook can be stated using this command:
$ ansible-playbook yum.yml -i inventory/lab

PLAY [all] ********************************************************************

GATHERING FACTS ***************************************************************
ok: [envtest]

TASK: [install vim] ***********************************************************
changed: [envtest]

PLAY RECAP ********************************************************************
envtest                    : ok=2    changed=1    unreachable=0    failed=0


Where -i is pointing to the inventory file and yum.yml is a name of file with playbook definition. Output contains a list of tasks executed and summary with number of successes, changes and failures. 

Ansible playbook and variables:

In the all previous examples, only has host has been used and there was no variables set. Let's add some more complexity to playbook and show how flexible is Ansible. Both variables defined for particular host and group will be used to create a text file based on template.

Template file ( Ansible supports Jinja2 template language)
$ cat template.j2
Host name is {{ ansible_hostname }}
Host var password is set to {{ password }}
Group var groupname is set to {{ groupname }}

Ansible playbook:
$ cat template.yml
---
- hosts: all
  tasks:
     - name: generate scripts
       template: src=template.j2 dest=/tmp/output mode=0644

Running playbook:
$ ansible-playbook -i inventory/lab template.yml

PLAY [all] ********************************************************************

GATHERING FACTS ***************************************************************
ok: [envtest]

TASK: [generate scripts] ******************************************************
changed: [envtest]

PLAY RECAP ********************************************************************
envtest                    : ok=2    changed=1    unreachable=0    failed=0
Output file on envtest server
$ ssh oracle@172.16.180.190
Last login: Tue Sep 29 12:29:59 2015 from 172.16.180.1
[oracle@envtest ~]$ cat /tmp/output
Host name is envtest
Host var password is set to host env password
Group var groupname is set to Test group
[oracle@envtest ~]$


If you think that Ansible is difficult at that stage, and there is no use for DBA's wait until next blog post when I will use a ansible-Oracle playbooks to install Oracle on server changing only one files with a variables. 

regards,
Marcin 


 

 

Sunday, September 20, 2015

Automation for DBA - Vagrant part 2

Hello,

In the last post I presented steps to create Virtual Machine using Vagrant with Virtual Box. It is a native combination but there are other possibilities as well. This post will cover an integration of Vagrant tool with KVM hypervisor and AWS EC2 cloud.

In both cases, an additional plugins has to be installed:
-    vagrant-libvirt plugin for KVM - https://github.com/pradels/vagrant-libvirt
-    vagrant-aws plugin for EC2 - https://github.com/mitchellh/vagrant-aws

If there is a requirement to use a Vagrant boxes defined for other hypervisors, there is a plugin called vagrant-mutate, which help you to convert it from one format to another.

KVM installation steps:


Host configuration:
-    Fedora 21
-    QEMU emulator version 2.1.


List of plugins is maintain on this page - Available-Vagrant-Plugins

Vagrant has very easy way to manage plugins. If plugin is listed on the official page installation process is limited to single command.

[pioro@piorovm]$ vagrant plugin install vagrant-mutate
Installing the 'vagrant-mutate' plugin. This can take a few minutes...
Installed the plugin 'vagrant-mutate (1.0.1)'!

[pioro@piorovm]$ vagrant plugin install vagrant-libvirt
Installing the 'vagrant-libvirt' plugin. This can take a few minutes...
Installed the plugin 'vagrant-libvirt (0.0.30)'!
Displaying a list of installed plugins with version.
[pioro@piorovm]$ vagrant plugin list
vagrant-libvirt (0.0.30)
vagrant-mutate (1.0.1)
vagrant-share (1.1.4)
As in the previous post a prebuild box racattack/oracle65 will be used. This box has been defined for Virtual Box so it has to be downloaded and migrated into KVM format. There is one caveat – after conversion new box will be still using a SATA drivers and not a native virtio drivers. If performance of Vagrant box is important, I will recommend creating a new native version of box for KVM configured with virtio.

Box racattack has two versions in official repository and mutate plugin was unable to use it directly. To workaround that issue a box has to be downloaded first using ex. wget. 
[pioro@piorovm ~]$ wget https://atlas.hashicorp.com/racattack/boxes/oracle65/versions/14.11.01/providers/virtualbox.box
--2015-08-19 23:33:57--  https://atlas.hashicorp.com/racattack/boxes/oracle65/versions/14.11.01/providers/virtualbox.box
Resolving atlas.hashicorp.com (atlas.hashicorp.com)... 107.23.224.212, 54.175.82.169
Connecting to atlas.hashicorp.com (atlas.hashicorp.com)|107.23.224.212|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://dl.dropbox.com/s/ohj0wjhrwuhowoi/oracle6-racattack.14.11.01.box?dl=0 [following]
--2015-08-19 23:33:58--  https://dl.dropbox.com/s/ohj0wjhrwuhowoi/oracle6-racattack.14.11.01.box?dl=0
Resolving dl.dropbox.com (dl.dropbox.com)... 54.243.87.222
Connecting to dl.dropbox.com (dl.dropbox.com)|54.243.87.222|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://dl.dropboxusercontent.com/s/ohj0wjhrwuhowoi/oracle6-racattack.14.11.01.box?dl=0 [following]
--2015-08-19 23:33:58--  https://dl.dropboxusercontent.com/s/ohj0wjhrwuhowoi/oracle6-racattack.14.11.01.box?dl=0
Resolving dl.dropboxusercontent.com (dl.dropboxusercontent.com)... 107.21.214.189, 107.22.170.202, 107.22.227.179, ...
Connecting to dl.dropboxusercontent.com (dl.dropboxusercontent.com)|107.21.214.189|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1185789020 (1.1G) [application/octet-stream]
Saving to: ‘virtualbox.box’
In the next steps it will be renamed, converted and added to box's repository.
[pioro@piorovm ~]$ mv virtualbox.box racattack.box

[pioro@piorovm ~]$ vagrant mutate racattack.box libvirt
Extracting box file to a temporary directory.
Converting racattack from virtualbox to libvirt.
    (100.00/100%)
Cleaning up temporary files.
The box racattack (libvirt) is now ready to use.

[pioro@piorovm ~]$ vagrant box list
homeora65 (libvirt, 0)
precise32 (virtualbox, 0)
racattack (libvirt, 0)
Vagrant configuration is ready and a new Virtual Machine can be created. Like in the previous host a new directory will be created and a Vagrant will be initialized. In the next step a VM will be started.
[pioro@piorovm ~]$ cd test
[pioro@piorovm test]$ vagrant init racattack
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
[pioro@piorovm test]$ vagrant up
Bringing machine 'default' up with 'libvirt' provider...
==> default: Uploading base box image as volume into libvirt storage...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default:  -- Name:              test_default
==> default:  -- Domain type:       kvm
==> default:  -- Cpus:              1
==> default:  -- Memory:            512M
==> default:  -- Base box:          racattack
==> default:  -- Storage pool:      default
==> default:  -- Image:             /VM/d1/images/test_default.img
==> default:  -- Volume Cache:      default
==> default:  -- Kernel:
==> default:  -- Initrd:
==> default:  -- Graphics Type:     vnc
==> default:  -- Graphics Port:     5900
==> default:  -- Graphics IP:       127.0.0.1
==> default:  -- Graphics Password: Not defined
==> default:  -- Video Type:        cirrus
==> default:  -- Video VRAM:        9216
==> default:  -- Keymap:            en-us
==> default:  -- Command line :
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...
==> default: Waiting for SSH to become available...
    default:
    default: Vagrant insecure key detected. Vagrant will automatically replace
    default: this with a newly generated keypair for better security.
    default:
    default: Inserting generated public key within guest...
    default: Removing insecure key from the guest if its present...
    default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Configuring and enabling network interfaces...
==> default: Rsyncing folder: /home/pioro/test/ => /vagrant
[pioro@piorovm test]$
There is a difference how files are synchronized between Vagrant and a VM machine in VirtualBox and other providers. Virtual Box is using a shared directory feature and KVM and EC2 are using rsync. Rsync process is started only once during VM start, and it to synchronize files after that “vagrant rsync” command has to be run

Like I mentioned in previous post, there are differences in Vagrant configuration between hypervisors. 

The following example is showing a configuration which will add two disks:
 - 10 GB none shared disk and 
 - 20 GB disk which can be shared between hosts. 

Memory size is set to 4 GB and number of vCPUs is set to 2. 
VM will have two network interfaces - one bridged to host network card (192.168.1.x) and one private KVM network (10.10.10.x)

Vagrant.configure(2) do |config|
config.vm.box = "racattack "

  config.vm.provider :libvirt do |libvirt|
 libvirt.storage_pool_name = "pool_d1"
        libvirt.memory=4096
        libvirt.cpus=2
        libvirt.storage :file, :size => '20G', :format => 'qcow2
        libvirt.storage :file, :size => '10G', :type => 'raw', :allow_existing => 'true', :bus=> 'scsi', :device=>'sda', :path=>'asmdisk1_2'
  end

  config.vm.define :host do |host|
         host.vm.hostname = "targetkvm"
         host.vm.network :public_network, :dev => "br0", :mode => "bridge", :type => "bridge", :ip => "192.168.1.152"
         host.vm.network :private_network, :libvirt__network_name => "private", :ip => "10.10.10.152"
  end

end
 
 
This is all about KVM for now but I believe this is a good starting point for experiments.

EC2 configuration

AWS integration with Vagrnat is based on plugin as well. The main difference is that Vagrant box is limited to a file definition and VM is based on AMI delivered by AWS.
This AMI image can be official AWS build, community one or the one created by you.

Installation of the additional plugin:

$ vagrant plugin install vagrant-aws
Installing the 'vagrant-aws' plugin. This can take a few minutes...
Installed the plugin 'vagrant-aws (0.6.0)'!
List of all plugins
$ vagrant plugin list
vagrant-aws (0.6.0)
vagrant-share (1.1.4, system)
Let's create a new directory and configure an EC2 container to start
$ mkdir awstest
$ cd awstest/
$ vagrant init
In the next step a VM has to be configured. This is example Vagrantfile
Vagrant.configure("2") do |config|

  # for AWS use dummy box 
  config.vm.box = "dummy"            

  
  config.vm.provider :aws do |aws, override|
    aws.access_key_id = "KEY_ID"
    aws.secret_access_key = "SECRET_ACCESS_KEY"
    aws.instance_type = "t1.micro"
    aws.security_groups = "quick-start-1"
    aws.keypair_name = "mykeys"
    aws.region = "eu-west-1"

    # this is official CentOS AMI
    # agree to CentOS license - http://wiki.centos.org/Cloud/AWS
    aws.ami = "ami-42718735"

    override.ssh.username = "root"
    override.ssh.private_key_path = "/tmp/mykey.pem"

    aws.block_device_mapping = [{ 'DeviceName' => '/dev/sdb1', 'Ebs.VolumeSize' => 50 }]
  end
end
Next step is easy "vagrant up" will start your EC2 instance
$ vagrant up
Bringing machine 'default' up with 'aws' provider...
==> default: Warning! The AWS provider doesn't support any of the Vagrant
==> default: high-level network configurations (`config.vm.network`). They
==> default: will be silently ignored.
==> default: Launching an instance with the following settings...
==> default:  -- Type: t1.micro
==> default:  -- AMI: ami-42718735
==> default:  -- Region: eu-west-1
==> default:  -- Keypair: mykeys
==> default:  -- Security Groups: ["quick-start-1"]
==> default:  -- Block Device Mapping: [{"DeviceName"=>"/dev/sdb1", "Ebs.VolumeSize"=>50}]
==> default:  -- Terminate On Shutdown: false
==> default:  -- Monitoring: false
==> default:  -- EBS optimized: false
==> default:  -- Assigning a public IP address in a VPC: false
==> default: Waiting for instance to become "ready"...
==> default: Waiting for SSH to become available...
==> default: Machine is booted and ready for use!
==> default: Rsyncing folder: /tmp/awstest/ => /vagrant
The following SSH command responded with a non-zero exit status.
Vagrant assumes that this means the command failed!

mkdir -p '/vagrant'

Stdout from the command:



Stderr from the command:

sudo: sorry, you must have a tty to run sudo
Instance is up and running. There was a problem with a rsync as there is problem with SUDO configuration in this AMI. It can be resolved but this is not an issue for this post. Let's try to ssh to EC2 VM now
$ vagrant ssh
[root@ip-10-105-167-95 ~]# ls -l /dev/x*
brw-rw----. 1 root disk 202, 64 Sep 20 10:43 /dev/xvde
brw-rw----. 1 root disk 202, 81 Sep 20 10:43 /dev/xvdf1
SSH to EC2 with Vagrant is very simple. You don't need to remember that is a public IP of EC2 instance, "vagrant ssh" will do a trick. 
To stop EC2 instance use "vagrant halt" and to delete EC2 instance use "vagrant destroy"

This is all I would like to share in this post. In the next one, I will use Vagrant with VirtualBox and I will show how to add an automatic Oracle installation to your VM.

regards,
Marcin
 

Wednesday, August 19, 2015

Automation for DBA - Vagrant part 1

Vagrant is an open source tool for building a virtual environment. It is based on text based configuration file and concept of prebuilt OS boxes.
One can find a lot of boxes on Atlas webpage (https://atlas.hashicorp.com/boxes/search), ready to download and use.  Although if there are any security concerns about using a OS from 3rd party webpage, I published a blog post about creating a new Vagrant box using Oracle Linux here and Oracle Virtual Box.

For simplicity of this series, predefined Linux box will be used in all examples.
There are two boxes that I’m using in my own work.
First one “kikitux/oracle6-racattack” created by Alvaro Miranda (https://github.com/kikitux) member of Rac Attack team. The other one is “racattack/oracle65” created by Jeremy Schneider who is also member of RacAttack team.

Vagrant itself is not a Virtualization provider. It can control different virtualization tools, including:
-    Virtual Box
-    AWS EC2 (with an additional plugin)
-    KVM (with an additional plugin)
-    VMWare Fusion / Workstation (paid version of Vagrant)

Most of the differences between providers are hidden by Vagrant workflow, and a configuration file defining a Vagrant box is similar between different providers.
Vagrant user is using same set of commands to start, stop or destroy virtual machine and does not have to know which provider is used for that.

There is a list of typical Vagrant commands:

-    vagrant init – creating a basic configuration file
-    vagrant up – starting a virtual machine
-    vagrant halt – stopping a virtual machine
-    vagrant destroy – delete a virtual machine
-    vagrant ssh – open a ssh session to vagrant user on virtual machine

After this short introduction let’s get started with a Vagrant on Laptop/Desktop (using Virtual Box as a provider)

Prerequisites:
-    vagrant software – https://www.vagrantup.com/downloads.html
-    Virtual Box - https://www.virtualbox.org/

In the first step a new directory will be created and used for Vagrant to initiate build of new virtual machine.


mkdir machine1
cd machine1

In the next step Vagrant machine will be initialized


vagrant init racattack/oracle65
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

A new Vagrantfile has been created with basic configuration. By default Vagrant is trying to change a default set of public / private key with a new pair generated on user machine. But there are some problem with that on latest version of Vagrant, so as a workaround please add this line to Vagrantfile (see https://github.com/mitchellh/vagrant/issues/5186)


config.ssh.insert_key = false

This is how Vagrantfile looks like without commented lines


Vagrant.configure(2) do |config|
    config.vm.box = "racattack/oracle65"
    config.ssh.insert_key = false
end

Now this machine can be started. If a box selected in an init stage is not existing yet in the provider catalog, it will be automatically downloaded and cloned to a new machine.

mprzepiorowski$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'racattack/oracle65' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'racattack/oracle65'
    default: URL: https://atlas.hashicorp.com/racattack/oracle65
==> default: Adding box 'racattack/oracle65' (v14.11.01) for provider: virtualbox
    default: Downloading: https://atlas.hashicorp.com/racattack/boxes/oracle65/versions/14.11.01/providers/virtualbox.box
==> default: Successfully added box 'racattack/oracle65' (v14.11.01) for 'virtualbox'!
==> default: Importing base box 'racattack/oracle65'...
==> default: Matching MAC address for NAT networking...
==> default: Checking if box 'racattack/oracle65' is up to date...
==> default: Setting the name of the VM: machine1_default_1439847872931_79029
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2222 (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Mounting shared folders...
    default: /vagrant => /Users/mprzepiorowski/Documents/machine1
Is this that simple? Yes it is – a new virtual machine with Oracle Linux is installed and ready. Next step is to ssh into a new created machine. There is no need to know neither a password nor an IP address,

mprzepiorowski$ vagrant ssh
[vagrant@oracle6-racattack ~]$ 

Now it is a time to customize this machine a little bit. Before that an existing machine has to be stopped using a Vagrant command


mprzepiorowski$ vagrant halt
==> default: Attempting graceful shutdown of VM...

As it was mentioned at the beginning of the post, Vagrant is using a text based configuration file. All customizations can be done by editing this file.
The configuration file structure can be divided into two sections.
First section defines a provider configuration and this part will change when a provider will be changed. Other section is provider independent and describes all other parameters.


Vagrant.configure(2) do |config|

# this part is common for all providers

  config.vm.box = "racattack/oracle65"
  config.ssh.insert_key = false
  config.vm.hostname = "machine1"

# provider specific configuration

  config.vm.provider "virtualbox" do |vb|
    vb.memory = 1024 
    vb.cpus = 2
  end

end

Starting a machine with a new configuration


mprzepiorowski$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'racattack/oracle65' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
==> default: Forwarding ports...
    default: 22 => 2222 (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Connection timeout. Retrying...
    default: Warning: Connection timeout. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Setting hostname...
==> default: Mounting shared folders...
    default: /vagrant => /Users/mprzepiorowski/Documents/machine1
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.

Checking changes


[vagrant@machine1 ~]$ free
             total       used       free     shared    buffers     cached
Mem:       1020696     121356     899340          0      10980      42100
-/+ buffers/cache:      68276     952420
Swap:      8191996          0    8191996
[vagrant@machine1 ~]$ cat /proc/cpuinfo | grep processor
processor : 0
processor : 1

All looks good there. Virtual machine has 1 GB of RAM and two vCPU. Machine name has been changed to machine1 as well.

The last change tested here will be a new disk assigned to virtual machine. The box used for an example already has a “sdb” device and an “u01” file system defined inside a box. In this case device “sdc” will be added to the box and can be configured later as a “u02” file system if necessary. The file system creation and configuration will be presented in the next posts about Ansible.

disk_filename = 'disk-u02.vdi'

Vagrant.configure(2) do |config|

# this part is common for all providers

  config.vm.box = "racattack/oracle65"
  config.ssh.insert_key = false
  config.vm.hostname = "machine1"

# provider specific configuration

  config.vm.provider "virtualbox" do |vb|
    vb.memory = 1024
    vb.cpus = 2
    unless File.exist?(disk_filename)
       vb.customize ['createhd', '--filename', disk_filename, '--size', (5*1024), '--variant', 'Standard']
    end
    vb.customize ['storageattach', :id, '--storagectl', 'SATA Controller', '--port', 2, '--device', 0, '--type', 'hdd', '--medium', disk_filename]
  end

end

Options specified in vb.customize are options of VBoxManage command of Virtual Box installation.

mprzepiorowski$ vagrant ssh
Last login: Mon Aug 17 22:01:36 2015 from 10.0.2.2
[vagrant@machine1 ~]$ ls -l /dev/sdc*
brw-rw---- 1 root disk 8, 32 Aug 17 22:25 /dev/sdc
[vagrant@machine1 ~]$ 

New disk is added and ready for other activities.

This is end of this post and I will present similar steps for KVM and EC2 virtual machines in the next one.

regards,
Marcin


Monday, August 10, 2015

Preparing a Oracle Linux for a Vagrant box

This post is an overview of an installation and configuration process of the Oracle Linux, which will be used as a machine “box” for Vagrant software using a Virtual Box platform.

Post itself is divided into two parts:

  1. Oracle Linux installation (points 1 to 12)
  2. Vagrant box configuration (points 13 to 16)

Part 1 – Oracle Linux installation

 

1. Create a new Virtual Machine - machine name will be used later to create Vagrant box

2. Set memory size - it does matter only for a installation process, as it can be changed later by Vagrant

3. Create a new disk, which will be used to keep a root and a swap file systems only. Any additional disks will be added dynamically via Vagrant. In this example, a disk size is set to 8 GB.





4. Mount Oracle Linux DVD into Virtual Machine drive


5. Configure 1st network interface to be NAT 

 


6. Add port forwarding for SSH



6.Start Virtual Machine and start a Oracle Linux installation. Some of the screens have been skipped in this post but a default values have been used for those ones.



7.Choose language and keyboard




8.Configure disk using a default basic definition



9.Configure a host name for a box, configure a network with DHCP and start up at boot time.


10.Default packages will be installed only at this time. Some will be added later.



11.Wait for process to be completed
 


12.Restart machine



Part 2 – Vagrant box configuration

13. Add Vagrant related configuration
  • Add user "vagrant" with password "vagrant"
  • Add Vagrant public insecure public key to .authorized_keys
  • Install Guest additional tools
Connect to VM using ssh
 
ssh -p 2022 root@localhost
The authenticity of host '[localhost]:2022 ([127.0.0.1]:2022)' can't be established.
RSA key fingerprint is 3e:4f:bd:ca:45:d6:e8:d4:6b:4d:02:bb:1f:c3:ad:a2.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:2022' (RSA) to the list of known hosts.
root@localhost's password:
Last login: Sat Jul 18 21:25:57 2015 from 10.0.2.2
Add user using password vagrant
[root@oelbox ~]# useradd vagrant
[root@oelbox ~]# passwd vagrant
Changing password for user vagrant.
New password:
BAD PASSWORD: it is based on a dictionary word
BAD PASSWORD: is too simple
Retype new password:
passwd: all authentication tokens updated successfully.
Add Vagrant public key
[root@oelbox ~]# su - vagrant
[vagrant@oelbox ~]$ mkdir .ssh
[vagrant@oelbox ~]$ chmod 700 .ssh
[vagrant@oelbox ~]$ cd .ssh
[vagrant@oelbox .ssh]$ wget --no-check-certificate https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub -O authorized_keys
--2015-07-18 21:29:16--  https://raw.githubusercontent.com/mitchellh/vagrant/master/keys/vagrant.pub
Resolving raw.githubusercontent.com... 23.235.44.133
Connecting to raw.githubusercontent.com|23.235.44.133|:443... connected.
WARNING: certificate common name “www.github.com” doesn’t match requested host name “raw.githubusercontent.com”.
HTTP request sent, awaiting response... 200 OK
Length: 409 [text/plain]
Saving to: “authorized_keys”

100%[===========================================================================================================================================================>] 409         --.-K/s   in 0s

2015-07-18 21:29:17 (94.1 MB/s) - “authorized_keys” saved [409/409]

[vagrant@oelbox .ssh]$ chmod 600 authorized_keys

Choose Install Guest Additional CD from Virtual Box menu and check if CDROM it was mounted

Install required packages and add Guest tools
[root@oelbox ~]# yum install kernel-uek-devel-3.8.13-16.2.1.el6uek.x86_64
[root@oelbox ~]# mount /dev/cdrom /mnt/
mount: block device /dev/sr0 is write-protected, mounting read-only
[root@oelbox ~]# /mnt/VBoxLinuxAdditions.run
Verifying archive integrity... All good.
Uncompressing VirtualBox 4.3.20 Guest Additions for Linux............
VirtualBox Guest Additions installer
Removing installed version 4.3.20 of VirtualBox Guest Additions...
Copying additional installer modules ...
Installing additional modules ...
Removing existing VirtualBox non-DKMS kernel modules       [  OK  ]
Building the VirtualBox Guest Additions kernel modules
Building the main Guest Additions module                   [  OK  ]
Building the shared folder support module                  [  OK  ]
Building the OpenGL support module                         [  OK  ]
Doing non-kernel setup of the Guest Additions              [  OK  ]
Starting the VirtualBox Guest Additions                    [  OK  ]
Installing the Window System drivers
Could not find the X.Org or XFree86 Window System, skipping. 

14. Install packages required for Oracle database and EPEL repository and Ansible. 
If you are using RedHat or Centos you can’t use oracle-rdbms packages. Oracle-ansible playbook, which will be used later will take care about all packages during a first installation.

[root@oelbox ~]# yum install oracle-rdbms-server-11gR2-preinstall.x86_64 oracle-rdbms-server-12cR1-preinstall.x86_64
[root@oelbox ~]# rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Retrieving http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
warning: /var/tmp/rpm-tmp.SCcj3K: Header V3 RSA/SHA256 Signature, key ID 0608b895: NOKEY
Preparing...                ########################################### [100%]
   1:epel-release           ########################################### [100%]
[root@oelbox ~]# yum install ansible

15. Configure sudo for vagrant user - disable requiretty and allow vagrant user to run sudo without password for all commands.
[root@oelbox ~]# visudo
#Defaults    requiretty
vagrant ALL=(ALL) NOPASSWD: ALL

16. Shutdown Virtual Machine.

This is end of configuration on the Linux level and this Virtual Machine can be used now as a source of the Vagrant box.


regards,
Marcin

Monday, July 20, 2015

How to avoid boring work

Over the past few years working as an IT consultant, I've learned that having easy access to a robust lab where I can learn or test solutions is critical. But my approach to building a test lab has changed over the years.
These days I prefer not to "build" a lab but to "define" one — it is a better use of my time.
Building requires a long step-by-step process of installing all the necessary components: OS, databases, app servers, one by one based on the documentation. 
Instead, I can define a lab using tools such as Ansible, Chef, Puppet, Oracle Enterprise Manager, on top of tools that manage Virtual Machines like Vagrant or libvirt.

I presented my thoughts during Irish Oracle User Group conference in March 2015 ( How to avoid boring workand now I would like to start a series of blog post about this.


Here is a list of topics I plan to cover in the next few weeks:
  • Creating and managing a Virtual Machine using Vagrant (create a new box, add storage, use KVM or AWS EC2 as providers )
  • Installing and configuring Oracle Database using Ansible ( Thanks to great work of Mikael Sandström on his Github Ansible repository )
  • Integration of Delphix "Data as a Service" product with an automated test environment build

regards,
Marcin