Virtualizing a physical linux, without VMware Converter, over the network, and quickly.

Note: Some Linux knowledge is assumed in this guide. After all, if you want to virtualize a linux I guess you know it to some degree.

[I didn't write this with the intention of it being a complete howto/manual, but rather to summarize the steps I took in case I have to do it again.]

I know it's a bit of a long title, but considering everything I found out there, I thought it was necessary to stress it a bit.

This method does NOT require Ghost, KVM, qemu, Virtual Box, vCenter, ESXi, etc.

I will only use normal Linux tools, and VMware Workstation as target platform to run my vm.

I won't install anything on the source OS. We want to keep it clean. So in order to read the disk I will use a live-cd distribution (could also be a live-usb distro).

Boot now from live cd on the source machine.

Now, to make an exact copy of the disk you just need the "dd" program.

With this you can copy the first disk without even needing to mount it.

# dd if=/dev/sda

you can copy it to another drive that you could have attached via usb
# dd if=/dev/sda of=/dev/sdb   (more info about this at the end [X])

to a file in a mounted external drive
# dd if=/dev/sda of=/media/mounted_external_drive/image_of_sda.img

But this would be VERY slow as it would copy the content bit by bit. You may want to add
bs=16M
to copy in chunks of 16MB
(you can read some performance comparisons out there)

and because you are reading from a place that may contain errors, this could come handy
conv=noerror,sync

You can also transfer the content over the network, piping to  ssh or to nc.

ssh Will encrypt the content, which will slow down the process and may not be needed if you are in a safe place. So I will pipe it to nc.

# dd if=/dev/sda conv=noerror,sync bs=16M | nc -q 3  192.168.1.88  19000

-q 3  Will make nc to stop 3 seconds after dd has stopped sending data
The 192.168.1.88 is the destination IP, and the 19000 (PORT) could be anything (better high and weird)

If you want to see the progress, you can put "pv" in the middle
# dd if=/dev/sda conv=noerror,sync bs=16M | pv | nc -q 3  192.168.1.88  19000

But if you don't have it, I will give you later an alternative.
In any case, when the sender is done, dd will give you an average speed for the transfer.


Now, on the receiver system there is some funky stuff.
My receiver is a Windows host, running workstation. In a vm I run the same linux live-cd and I give it a preallocated virtual disk a bit larger than the source physical drive. When this vm boots, I format that vdisk as ntfs and mount it. This vdisk will be a temporal container for the vdisk of the source physical machine (yes, vdisk within vdiks). I format it as ntfs because later we will mount this vmdk on windows and extract the image.

So on the receiver linux live-cd I run:
# nc -l -p 19000 > /mnt/ntfs_drive/source_physical_drive.img

If you want to see the progress of the transfer you can put pv in the middle
# nc -l -p 19000 | pv  > /mnt/ntfs_drive/source_physical_drive.img

OR you can open another console and run
# watch -n 20 ls -lh /mnt/ntfs_drive/
Which will run "ls -lh /mnt/ntfs_drive/" every 20 seconds, and you will be able to see the img file growing fat.



When the transfer is completed, you can unmount /mnt/ntfs_drive/ and shutdown the receiver Linux.

Now, from workstation, map/mount the vmdk that receiver Linux was writing into. Once it is mounted you will see the file source_physical_drive.img inside it.

Copy that file to some place inside the Windows filesystem. Then you can unmap/unmount that vmdk you just mounted.

Now, that file is the content of the source disk, but not in a format that workstation can use, but almost.

We just need to recreate the vmdk descriptor for that image file.

Using msdos, get the size of the img file (command dir on the location of the file)

Lets say it is 95066058752   (bytes)

Now divide that by 1024, and you will have the size in KB,  92837948

Create now a vdisk drive with the same size and the format you desire.

In my case I want:
-t 2                 preallocated single file
-c                   create
-a lsilogic       type
-s 92837948KB    size in KB


So I run
vmware-vdiskmanager -c -t 2 -a lsilogic -s 92837948KB  myVM.vmdk

On the folder you will have:
myVM.vmdk
myVM-flat.vmdk
source_physical_drive.img

Now you can remove myVM-flat.vmdk and rename source_physical_drive.img to myVM-flat.vmdk

You have now the content of your source physical machine in a vdisk format that Workstation can use.

You just need to create a VM and attach this vmdk to it.

I am writing these lines from a Linux that has been running for maaaany years on a Dell Inspiron and now is running on Workstation on another Dell Inspiron some days old.



Notes:
- If you are doing this over wifi, DON'T. You will kill the network. Connect both machines with cable.
- The destination Linux could be running from iso, you don't really need a physical CD. Any knoppix will do the job if you don't know what to use.
- Backup the source system before you start
- Disclaimer: Use this info at your own risk.
- I would normally copy/paste all the commands from the real execution instead of writing them by memory, but due to the circumstances that is not possible. With that being said, I have done my best to avoid any mistake. If you find any please let me know and I will update it.
- To improve speed/performance, connect the receiver linux to the network in workstation-bridge-mode (without replicating network state).


If you want to test the whole procedure but quickly, do this:
- In workstation, install a tiny linux in a vm
- Boot that vm from the linux-live-cd
- Create a 2nd vm with a vdisk and boot it from live-cd
- Do the transfer between both vms and then test the file extraction and conversion. When the transfer is completed you should have the same tiny linux running on 2 vms.
I used puppy linux for this test.
This experiment should not impact the network as all the networking is inside workstation (if you are using NAT connection).

------------------------

[X] If you want to copy from disk to disk and then be able to reshape the size of the partitions (destination may be larger than source and you want to use it all), there is a handy tool that will do all the hard work for you.

It is the gparted live CD. http://gparted.sourceforge.net/index.php

You boot from it, and you can clone partitions from one disk to another (both connected to the computer).

How to retrieve content using curl / wget passing through VMware Horizon authentication.

wget and curl are two programs commonly used in command line in Linux, but they can also be used in Windows.

Once you know how, it becomes really simple. The caveat is that sometimes it requires a click and paste.

Instructions:
1. Install cliget addon for firefox
2. If you want to use "wget" you don't have to do anything. If you want to use "curl" go to about:config in firefox and change cliget.wget to false and cliget.curl to true
3. Using your firefox browser navigate to the page were there is a link to the content you want to retrieve or the page itself that you want to retrieve. Right click over the link or the page and select "copy curl/wget for link" or "copy curl/wget for page".
4. Go to the command line and paste

And you shall get the content you asked for, via command line.

Note that you can run that command on ANY computer, not just the computer you were running Firefox on. The cookies and session ids are normally temporal, so they will be valid for a while only.

Note as well that with curl you can use wild-cards. Example:    http://myweb.com/img[0-23].jpg

Easy generation of configuration file for Pools with multiple network labels

One of the new features of VMware Horizon View 5.2 allows you to create pools with different portgroups or network labels.

As part of the configuration process you have to generate a configuration file, and for that you need to run a command with lots of parameters.

I found a way to generate most of the difficult parameters, so you don't spend half an hour (or more) trying different combinations of parameters.

All the info below.


All you have to do is to copy & paste the function below in a "View PowerCLI" console (not the same as "PowerCLI" alone) and then run the command GenerateNetworkConfFile

I include some execution examples below.

If you have issues or suggestions please submit a comment.


#====== SOURCE CODE ============================================================================

function GenerateNetworkConfFile {

# Copyright: Ruben Miguelez Garcia @ 2013 -> http://vmutils.blogspot.com

# Log in to VC if not already done
echo ""
$loggedinvc = read-host -Prompt "Have you already logged into the vCenter used by View using the command Connect-VIServer ?  [y]/n"
if ($loggedinvc -eq "n"){ Connect-VIServer };

# Request a few details
echo ""
$vmname= read-host -Prompt "What is the name of the VM you want to generate a network configuration file for?"


echo "";
$MaxVMsPerNetworkLabel= read-host -Prompt "How many IPs do you want to set up as default limit per network label? [240]"
if ($MaxVMsPerNetworkLabel -eq "") { $MaxVMsPerNetworkLabel="240"; }


# The two functions below come from the View PowerCLI documentation (Integration guide)
# The 1st function has a small modification
#------------------------------------------
function Get-SnapshotPath {
 param(
  $Snapshot
 )
 PROCESS {
  if ($Snapshot) {
   $SnapPath = $Snapshot.Name
   do {
    $SnapPath = "$($snapshot.Parent.Name)/" + $SnapPath
    $Snapshot = $Snapshot.Parentsnapshot
   } while ($Snapshot.Parent -ne $null)

   #$SnapPath = "/" + $SnapPath
   if ($SnapPath[0] -ne "/") { $SnapPath = "/" + $SnapPath }
   $SnapPath
  } Else {
   Write-Error "Snapshot not found"
  }
 }
}
#------------------------------------------
function VVGetPath($InvObject){
    if($InvObject){
 
        $objectType = $InvObject.GetType().Name
        $objectBaseType = $InvObject.GetType().BaseType.Name
        if($objectType.Contains("DatastoreImpl")){
            Write-Error "Use the VVGetDataStorePath function to determine datastore paths."
            break
        }
        if(-not ($objectBaseType.Contains("InventoryItemImpl") -or $objectBaseType.Contains("FolderImpl") -or $objectBaseType.Contains("DatacenterImpl") -or $objectBaseType.Contains("VMHostImpl") ) ){
            Write-Error ("The provided object is not an expected vSphere object type. Object type is " + $objectType)
            break
        }
 
        $path = ""
        # Recursively move up through the inventory hierarchy by parent or folder.
        if($InvObject.ParentId){
            $path = VVGetPath(Get-Inventory -Id $InvObject.ParentId)
        } elseif ($InvObject.FolderId){
            $path = VVGetPath(Get-Folder -Id $InvObject.FolderId)
        }
 
        # Build the path, omitting the "Datacenters" folder at the root.
        if(-not $InvObject.isChildTypeDatacenter){ # Add object to the path.
            $path = $path + "/" + $InvObject.Name
        }
        $path
    }
}
#------------------------------------------


# Get the rest of the details needed automatically
$vm=get-vm -name $vmname
$vc_id=(Get-ViewVC).vc_id ;
$ClusterPath =   "/" + (Get-Datacenter -vm $vm ).name + "/host/" + (get-cluster -vm $vm)
$ParentVmPath = vvgetpath($vm) ;

# Use [0] to get the name of 1st snapshot, [1] for the 2nd, and so on. [-1] for the last one.
# This may not work well if there are multiple snapshot branches.
# The if/else is there to account for cases where there is 1 or multiple snapshots
# If only one, we take that, if multiple, we take the last one
$snaps=$vm | get-snapshot
if ($snaps.gettype().tostring() -eq "VMware.VimAutomation.ViCore.Impl.V1.VM.SnapshotImpl")
 { $ParentSnapshotPath= Get-SnapshotPath ($snaps) }
 else { $ParentSnapshotPath= Get-SnapshotPath ($snaps)[-1] }



# The file will be placed on C:\
$NetworkLabelConfigFile= "C:\" + $vm.name + "NetSpec.txt" ;

# Display what we got  and verify they have valid values
echo ""
echo "Here are the details collected:"
echo "vc_id = $vc_id" ; # Example value: e53c44c5-d5a6-43ec-bab4-f91f7f3fa91e
echo "ClusterPath = $ClusterPath" ; # Example value: /Training/host/ViewCluster
echo "ParentVmPath=$ParentVmPath "; # Example value: /Training/vm/Windows8
echo "ParentSnapshotPath=$ParentSnapshotPath "; # Example value: /Base1/With License/With perf improv/end
echo "NetworkLabelConfigFile=$NetworkLabelConfigFile "; # Example value: C:\Windows8NetSpec.txt


# Generate the configuration file
Export-NetworkLabelSpecForLinkedClone `
-Vc_id $vc_id `
-ClusterPath $ClusterPath `
-ParentVmPath $ParentVmPath `
-ParentSnapshotPath $ParentSnapshotPath `
-MaxVMsPerNetworkLabel $MaxVMsPerNetworkLabel `
-NetworkLabelConfigFile $NetworkLabelConfigFile

}
# End of GenerateNetworkConfFile


#====== USAGE EXAMPLES ============================================================================



######### Using default values in 2 questions

> GenerateNetworkConfFile

Have you already logged into the vCenter used by View using the command Connect-VIServer ?  [y]/n:

What is the name of the VM you want to generate a network configuration file for?: Windows8_empty


How many IPs do you want to set up as default limit per network label? [240]:


Here are the details collected:
vc_id = e53c44c5-d5a6-43ec-bab4-f91f7f3fa91e
ClusterPath = /Training/host/ViewCluster
ParentVmPath=/Training/vm/Windows8_empty
ParentSnapshotPath=/Base Line
NetworkLabelConfigFile=C:\Windows8_emptyNetSpec.txt




######## With problem because VM has no snapshots


PS C:\Users\training> GenerateNetworkConfFile

Have you already logged into the vCenter used by View using the command Connect-VIServer ?  [y]/n: n

cmdlet Connect-VIServer at command pipeline position 1
Supply values for the following parameters:
Server[0]: vc.company.com
Server[1]:

Name                           Port  User
----                           ----  ----
vc.company.com           443   admin

What is the name of the VM you want to generate a network configuration file for?: VM_HW_v9


How many IPs do you want to set up as default limit per network label? [240]:

You cannot call a method on a null-valued expression.
At line:74 char:19
+ if ($snaps.gettype <<<< ().tostring() -eq "VMware.VimAutomation.ViCore.Impl.V1.VM.SnapshotImpl")
    + CategoryInfo          : InvalidOperation: (gettype:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull


Here are the details collected:
vc_id = e53c44c5-d5a6-43ec-bab4-f91f7f3fa91e
ClusterPath = /Training/host/ViewCluster
ParentVmPath=/Training/vm/VM_HW_v9
ParentSnapshotPath=/snap1/snap2
NetworkLabelConfigFile=C:\VM_HW_v9NetSpec.txt
Export-NetworkLabelSpecForLinkedClone : PowershellService::ExportNetworkLabelSpecForLinkedClone FAILED, error=No NIC found f
or base VM: /Training/vm/VM_HW_v9 and snapshot: /snap1/snap2
At line:88 char:38
+ Export-NetworkLabelSpecForLinkedClone <<<<  `
    + CategoryInfo          : InvalidResult: (vmware.view.pow...cForLinkedClone:ExportNetworkLabelSpecForLinkedClone) [Expo
   rt-NetworkLabelSpecForLinkedClone], Exception
    + FullyQualifiedErrorId : PowershellService::ExportNetworkLabelSpecForLinkedClone FAILED,vmware.view.powershell.cmdlets
   .ExportNetworkLabelSpecForLinkedClone



############ Another example

> GenerateNetworkConfFile

Have you already logged into the vCenter used by View using the command Connect-VIServer ?  [y]/n:

What is the name of the VM you want to generate a network configuration file for?: Windows7


How many IPs do you want to set up as default limit per network label? [240]: 999


Here are the details collected:
vc_id = e53c44c5-d5a6-43ec-bab4-f91f7f3fa91e
ClusterPath = /Training/host/ViewCluster
ParentVmPath=/Training/vm/Windows7
ParentSnapshotPath=/Base Line
NetworkLabelConfigFile=C:\Windows7NetSpec.txt




################ Checking the output file from console


> cat C:\XP_tiny_emptyNetSpec.txt
#Network Label Configuration Spec

#WARNING! Setting enabled flag to false will
#turn off the automatic network label assignment
#for newly provisioned desktops.
enabled=true

#Parameter Definition for NIC
nic1=Network adapter 1

#Parameter Definition for Network
network01=VDI_Net1
network02=VDI_Net2
network03=VDI_Net3
network04=VM Network
network05=VM Network 2

#Network Label Attribute Definition
#Expected format:
#..maxvm=

####nic1.network01.maxvm=240
####nic1.network02.maxvm=240
####nic1.network03.maxvm=240
####nic1.network04.maxvm=240
####nic1.network05.maxvm=240

Using gatewayInsertExtract PowerCLI script