Vagrant re-packaging and ssh
Many of us use Vagrant from Hashicorp to create and keep reference images of VMs for different purposes like a test or for fast deployment. Sometimes these images need to be refreshed or modified. Several days ago I helped to fix ssh issue for a VM created from a new re-packaged Vagrant box. The new VM created from the Vagrant box had a problem with ssh connection from Vagrant interface. I will try to explain the cause of the problem and demonstrate how to fix it.
So, you got the basic host from the vagrant cloud, added it to your local repository and fired a new VM. Everything works fine. You can start a new VM without any problem and connect using the "vagrant ssh" command.
Glebs-MacBook:~ otochkin$ vagrant box list
generic/oracle7 (virtualbox, 1.2.32)
Glebs-MacBook:VirtualBoxVMs otochkin$ cd ol74.1/
Glebs-MacBook:ol74.1 otochkin$ ll
total 8
-rw-r--r-- 1 otochkin staff 3022 8 Nov 11:46 Vagrantfile
Glebs-MacBook:ol74.1 otochkin$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'generic/oracle7' is up to date...
==> default: A newer version of the box 'generic/oracle7' is available! You currently
==> default: have version '1.2.32'. The latest is version '1.3.28'. Run
==> default: `vagrant box update` to update.
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
==> default: Forwarding ports...
default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Remote connection disconnect. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
default: No guest additions were detected on the base box for this VM! Guest
default: additions are required for forwarded ports, shared folders, host only
default: networking, and more. If SSH fails on this machine, please install
default: the guest additions and repackage the box to continue.
default:
default: This is not an error message; everything may continue to work properly,
default: in which case you may ignore this message.
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.
Glebs-MacBook:ol74.1 otochkin$ vagrant box list
generic/oracle7 (virtualbox, 1.2.32)
Glebs-MacBook:ol74.1 otochkin$ vagrant ssh
Last login: Thu Nov 9 10:21:06 2017 from gateway
[vagrant@bazinga ~]$
Now you decide to customize it adding some packages, users or maybe doing something else. Let's say you have updated OS on the box and added the oracle preinstall rpm to the system.
[root@bazinga ~]# yum update
Loaded plugins: ulninfo
ol7_UEKR4 | 1.2 kB 00:00:00
ol7_latest | 1.4 kB 00:00:00
...
Complete!
[root@bazinga ~]#
[root@bazinga ~]# yum install oracle-rdbms-server-12cR1-preinstall.x86_64
Loaded plugins: ulninfo
Resolving Dependencies
--> Running transaction check
---> Package oracle-rdbms-server-12cR1-preinstall.x86_64 0:1.0-6.el7 will be installed
...
Complete!
[root@bazinga ~]#
Now it makes perfect sense to save the modified box to reuse it again without updating every time. To do so we re-package the Vagrant box. It looks pretty easy, you just need to find your VM name and run the command "vagrant package ..."
Glebs-MacBook:~ otochkin$ VBoxManage list vms
"ol741_default_1510174445179_51351" {41f54c19-3c2e-468f-9810-447f0256b483}
Glebs-MacBook:~ otochkin$ cd VirtualBoxVMs/ol741_default_1510174445179_51351/
Glebs-MacBook:ol741_default_1510174445179_51351 otochkin$ vagrant package --base ol741_default_1510174445179_51351
==> ol741_default_1510174445179_51351: Attempting graceful shutdown of VM...
ol741_default_1510174445179_51351: Guest communication could not be established! This is usually because
ol741_default_1510174445179_51351: SSH is not running, the authentication information was changed,
ol741_default_1510174445179_51351: or some other networking issue. Vagrant will force halt, if
ol741_default_1510174445179_51351: capable.
==> ol741_default_1510174445179_51351: Forcing shutdown of VM...
==> ol741_default_1510174445179_51351: Clearing any previously set forwarded ports...
==> ol741_default_1510174445179_51351: Exporting VM...
==> ol741_default_1510174445179_51351: Compressing package to: /Users/otochkin/VirtualBoxVMs/ol741_default_1510174445179_51351/package.box
Glebs-MacBook:ol741_default_1510174445179_51351 otochkin$ ll
total 9496896
drwx------ 5 otochkin staff 170 28 Dec 11:27 Logs
-rw------- 1 otochkin staff 3444768768 28 Dec 11:39 generic-oracle7-virtualbox-disk001.vmdk
-rw------- 1 otochkin staff 2503 28 Dec 11:39 ol741_default_1510174445179_51351.vbox
-rw------- 1 otochkin staff 2600 28 Dec 11:39 ol741_default_1510174445179_51351.vbox-prev
-rw-r--r-- 1 otochkin staff 1417630033 28 Dec 11:45 package.box
Glebs-MacBook:ol741_default_1510174445179_51351 otochkin$
So far so good and we are adding the packaged box to our local repository.
Glebs-MacBook:ol741_default_1510174445179_51351 otochkin$ vagrant box add ol741 package.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'ol741' (v0) for provider:
box: Unpacking necessary files from: file:///Users/otochkin/VirtualBoxVMs/ol741_default_1510174445179_51351/package.box
==> box: Successfully added box 'ol741' (v0) for 'virtualbox'!
Glebs-MacBook:ol741_default_1510174445179_51351 otochkin$ vagrant box list
generic/oracle7 (virtualbox, 1.2.32)
ol741 (virtualbox, 0)
Glebs-MacBook:ol741_default_1510174445179_51351 otochkin$
Now we can try to use the new image and fire a new VM.
Glebs-MacBook:ol741 otochkin$ vagrant init ol741
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
Glebs-MacBook:ol741 otochkin$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ol741'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: ol741_default_1514479772110_47964
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
==> default: Forwarding ports...
default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default: Warning: Authentication failure. Retrying...
default: Warning: Authentication failure. Retrying...
default: Warning: Authentication failure. Retrying...
default: Warning: Authentication failure. Retrying...
default: Warning: Authentication failure. Retrying...
...
What happened? Looks like the default key based ssh authentication doesn't work anymore. Let's have a look at our vagrant ssh config.
Glebs-MacBook:ol741 otochkin$ vagrant ssh-config
Host default
HostName 127.0.0.1
User vagrant
Port 2222
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile /Users/otochkin/.vagrant.d/insecure_private_key
IdentitiesOnly yes
LogLevel FATAL
Glebs-MacBook:ol741 otochkin$
It looks like we are using a default standard "insecure" ssh key and it doesn't work for our new VM. Now we have to verify what we have in the newly created VM. The key-based authentication doesn't work but we are still able to connect using the password. When you run "vagrant ssh" it is trying to connect to user "vagrant" and we can use the password for the user. For base boxes downloaded from the Vagrant cloud, the default password in most cases is "vagrant".
Glebs-MacBook:ol74 otochkin$ vagrant ssh
vagrant@127.0.0.1's password:
Last login: Thu Dec 28 08:29:37 2017 from gateway
[vagrant@bazinga ~]$
It works and we are in. It is good to know that losing the key is not the end of the world if you know the password. Let's check what we have in the "~/.ssh/authorized_keys file"
[vagrant@bazinga ~]$ cat .ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0YRAhPR+hfm9C5XV4VvT/CJ1ugiXIDC4+YNf6mDQMuP9U7LJkWZi5EFqsvBRKX......WaZJ vagrant
and compare with our public key for our "insecure_private_key" file.
Glebs-MacBook:ol74 otochkin$ ssh-keygen -y -f /Users/otochkin/.vagrant.d/insecure_private_key
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6NF8iallvQVp22WDkTkyrtvp9eWW6A8YVr+kz4TjGYe7gHzIw+niNltGEFHzD8+v1......WhQ==
Apparently, they are different and it is the root cause of the problem. As a workaround, we can just add the proper public key to the "~/.ssh/authorized_keys file on our VM and it will fix the problem for that particular VM.
[vagrant@bazinga ~]$ vi .ssh/authorized_keys
[vagrant@bazinga ~]$ logout
Connection to 127.0.0.1 closed.
Glebs-MacBook:ol74 otochkin$ vagrant ssh
Last login: Fri Dec 29 06:58:34 2017 from gateway
[vagrant@bazinga ~]$
But the workaround solution has a couple of drawbacks. At first, it is not secure since we are using the default key. And in a second, we will have the same ssh problem with every new VM created from the Vagrant box. We have to fix it permanently. The answer is really simple. Before packaging your box you have to add public key for your default insecure key to the authorized_keys file for your vagrant user on the box. Let's re-package the box and see how it works.
Glebs-MacBook:.vagrant.d otochkin$ VBoxManage list vms
"ol74_default_1514566114661_50519" {b7c603a1-f2e5-4e92-888e-21218cdaa519}
Glebs-MacBook:.vagrant.d otochkin$
Glebs-MacBook:ol74 otochkin$ vagrant package --base ol74_default_1514566114661_50519
==> ol74_default_1514566114661_50519: Attempting graceful shutdown of VM...
==> ol74_default_1514566114661_50519: Clearing any previously set forwarded ports...
==> ol74_default_1514566114661_50519: Exporting VM...
==> ol74_default_1514566114661_50519: Compressing package to: /Users/otochkin/VirtualBoxVMs/ol74/package.box
Glebs-MacBook:ol74 otochkin$
Adding the box to the local repository.
Glebs-MacBook:ol74 otochkin$ vagrant box add ol74 package.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'ol74' (v0) for provider:
box: Unpacking necessary files from: file:///Users/otochkin/VirtualBoxVMs/ol74/package.box
==> box: Successfully added box 'ol74' (v0) for 'virtualbox'!
Glebs-MacBook:ol74 otochkin$ vagrant box list
generic/oracle7 (virtualbox, 1.2.32)
ol74 (virtualbox, 0)
ol741 (virtualbox, 0)
Glebs-MacBook:ol74 otochkin$
And try to fire the new VM using our new packaged Vagrant box.
Glebs-MacBook:ol74 otochkin$ vagrant init ol74
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.
Glebs-MacBook:ol74 otochkin$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ol74'...
==> default: Matching MAC address for NAT networking...
==> default: Setting the name of the VM: ol74_default_1514567764721_10259
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
default: Adapter 1: nat
==> default: Forwarding ports...
default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
default: SSH address: 127.0.0.1:2222
default: SSH username: vagrant
default: SSH auth method: private key
default:
default: Vagrant insecure key detected. Vagrant will automatically replace
default: this with a newly generated keypair for better security.
default:
default: Inserting generated public key within guest...
default: Removing insecure key from the guest if it's present...
default: Key inserted! Disconnecting and reconnecting using new SSH key...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
default: No guest additions were detected on the base box for this VM! Guest
default: additions are required for forwarded ports, shared folders, host only
default: networking, and more. If SSH fails on this machine, please install
default: the guest additions and repackage the box to continue.
default:
default: This is not an error message; everything may continue to work properly,
default: in which case you may ignore this message.
==> default: Setting hostname...
Glebs-MacBook:ol74 otochkin$
Glebs-MacBook:ol74 otochkin$ vagrant ssh
Last login: Fri Dec 29 08:53:41 2017 from gateway
[vagrant@bazinga ~]$
It works now. Vagrant has replaced the insecure key with a new one and added the new key to the configuration for the new VM. You can see that the vagrant is using the default key for initial connection and process of replacing the old key with a new one. So it resolves both of our problems.
I hope that this small post can help a few people to save time and use the Vagrant more.
