gatewayInsertExtract v3

[UPDATE NOTE: The work creating, providing and supporting scripts for ESX/ESXi/vCenter has been discontinued. Apologies for the inconvenience.]
Script to insert or extract Gateway vms into the linked-clone Pods.

Note this script only operates with vms, the Operating System inside the vm as well as its configuration is up to you.

The gateway vms are there to provide external connectivity to the vms inside the Pod. Without it you could use the Remote Console to access each one of the vms but you may not want your students to be using the vSphere (Web) Client.

Those Gateway vms would have at least 2 NICs, one connected to the Pod and one to the external network from where the students can access their own vms.

The reason to extract them is because each one may contain some networking configuration that may be reused across training sessions. So before deleting the Pods, you extract them to preserve them. And when the Pods for the new training are ready you insert them back into the pods. This works well whether the Pods are in a DRS-cluster or in standalone hosts, as far as the Gateway vms were created in similar conditions.


First version (outdated)
https://www.youtube.com/watch?v=hf9ZO0atjNo

The second version has less parameters and its easier to use.
It is used in this video together with other scripts https://www.youtube.com/watch?v=ZAqGE6sROkU

A third version allows to insert/extract the Gateway vms into Pods of different names.
http://www.youtube.com/watch?v=yF1tr-YZqNw



Some clients use Monowall [1] for the Gateway vm and pfSense [2] for the outer router. The options, combinations and tools are endless.

[1] http://m0n0.ch/wall/
[2] www.pfsense.org



Together with this, I provide another script for doing NIC connectivity changes over a number of vms.

Some clients had complex setups where one NIC in specific vms had to be connected to a singular network. With this script you can modify the NIC connectivity of those vms in one go.

http://www.youtube.com/watch?v=3owJvyOUveA

Virtualizing a physical linux, without VMware Converter, over the network, and quickly.

Note: Some Linux knowledge is assumed in this guide. After all, if you want to virtualize a linux I guess you know it to some degree.

[I didn't write this with the intention of it being a complete howto/manual, but rather to summarize the steps I took in case I have to do it again.]

I know it's a bit of a long title, but considering everything I found out there, I thought it was necessary to stress it a bit.

This method does NOT require Ghost, KVM, qemu, Virtual Box, vCenter, ESXi, etc.

I will only use normal Linux tools, and VMware Workstation as target platform to run my vm.

I won't install anything on the source OS. We want to keep it clean. So in order to read the disk I will use a live-cd distribution (could also be a live-usb distro).

Boot now from live cd on the source machine.

Now, to make an exact copy of the disk you just need the "dd" program.

With this you can copy the first disk without even needing to mount it.

# dd if=/dev/sda

you can copy it to another drive that you could have attached via usb
# dd if=/dev/sda of=/dev/sdb   (more info about this at the end [X])

to a file in a mounted external drive
# dd if=/dev/sda of=/media/mounted_external_drive/image_of_sda.img

But this would be VERY slow as it would copy the content bit by bit. You may want to add
bs=16M
to copy in chunks of 16MB
(you can read some performance comparisons out there)

and because you are reading from a place that may contain errors, this could come handy
conv=noerror,sync

You can also transfer the content over the network, piping to  ssh or to nc.

ssh Will encrypt the content, which will slow down the process and may not be needed if you are in a safe place. So I will pipe it to nc.

# dd if=/dev/sda conv=noerror,sync bs=16M | nc -q 3  192.168.1.88  19000

-q 3  Will make nc to stop 3 seconds after dd has stopped sending data
The 192.168.1.88 is the destination IP, and the 19000 (PORT) could be anything (better high and weird)

If you want to see the progress, you can put "pv" in the middle
# dd if=/dev/sda conv=noerror,sync bs=16M | pv | nc -q 3  192.168.1.88  19000

But if you don't have it, I will give you later an alternative.
In any case, when the sender is done, dd will give you an average speed for the transfer.


Now, on the receiver system there is some funky stuff.
My receiver is a Windows host, running workstation. In a vm I run the same linux live-cd and I give it a preallocated virtual disk a bit larger than the source physical drive. When this vm boots, I format that vdisk as ntfs and mount it. This vdisk will be a temporal container for the vdisk of the source physical machine (yes, vdisk within vdiks). I format it as ntfs because later we will mount this vmdk on windows and extract the image.

So on the receiver linux live-cd I run:
# nc -l -p 19000 > /mnt/ntfs_drive/source_physical_drive.img

If you want to see the progress of the transfer you can put pv in the middle
# nc -l -p 19000 | pv  > /mnt/ntfs_drive/source_physical_drive.img

OR you can open another console and run
# watch -n 20 ls -lh /mnt/ntfs_drive/
Which will run "ls -lh /mnt/ntfs_drive/" every 20 seconds, and you will be able to see the img file growing fat.



When the transfer is completed, you can unmount /mnt/ntfs_drive/ and shutdown the receiver Linux.

Now, from workstation, map/mount the vmdk that receiver Linux was writing into. Once it is mounted you will see the file source_physical_drive.img inside it.

Copy that file to some place inside the Windows filesystem. Then you can unmap/unmount that vmdk you just mounted.

Now, that file is the content of the source disk, but not in a format that workstation can use, but almost.

We just need to recreate the vmdk descriptor for that image file.

Using msdos, get the size of the img file (command dir on the location of the file)

Lets say it is 95066058752   (bytes)

Now divide that by 1024, and you will have the size in KB,  92837948

Create now a vdisk drive with the same size and the format you desire.

In my case I want:
-t 2                 preallocated single file
-c                   create
-a lsilogic       type
-s 92837948KB    size in KB


So I run
vmware-vdiskmanager -c -t 2 -a lsilogic -s 92837948KB  myVM.vmdk

On the folder you will have:
myVM.vmdk
myVM-flat.vmdk
source_physical_drive.img

Now you can remove myVM-flat.vmdk and rename source_physical_drive.img to myVM-flat.vmdk

You have now the content of your source physical machine in a vdisk format that Workstation can use.

You just need to create a VM and attach this vmdk to it.

I am writing these lines from a Linux that has been running for maaaany years on a Dell Inspiron and now is running on Workstation on another Dell Inspiron some days old.



Notes:
- If you are doing this over wifi, DON'T. You will kill the network. Connect both machines with cable.
- The destination Linux could be running from iso, you don't really need a physical CD. Any knoppix will do the job if you don't know what to use.
- Backup the source system before you start
- Disclaimer: Use this info at your own risk.
- I would normally copy/paste all the commands from the real execution instead of writing them by memory, but due to the circumstances that is not possible. With that being said, I have done my best to avoid any mistake. If you find any please let me know and I will update it.
- To improve speed/performance, connect the receiver linux to the network in workstation-bridge-mode (without replicating network state).


If you want to test the whole procedure but quickly, do this:
- In workstation, install a tiny linux in a vm
- Boot that vm from the linux-live-cd
- Create a 2nd vm with a vdisk and boot it from live-cd
- Do the transfer between both vms and then test the file extraction and conversion. When the transfer is completed you should have the same tiny linux running on 2 vms.
I used puppy linux for this test.
This experiment should not impact the network as all the networking is inside workstation (if you are using NAT connection).

------------------------

[X] If you want to copy from disk to disk and then be able to reshape the size of the partitions (destination may be larger than source and you want to use it all), there is a handy tool that will do all the hard work for you.

It is the gparted live CD. http://gparted.sourceforge.net/index.php

You boot from it, and you can clone partitions from one disk to another (both connected to the computer).

How to retrieve content using curl / wget passing through VMware Horizon authentication.

wget and curl are two programs commonly used in command line in Linux, but they can also be used in Windows.

Once you know how, it becomes really simple. The caveat is that sometimes it requires a click and paste.

Instructions:
1. Install cliget addon for firefox
2. If you want to use "wget" you don't have to do anything. If you want to use "curl" go to about:config in firefox and change cliget.wget to false and cliget.curl to true
3. Using your firefox browser navigate to the page were there is a link to the content you want to retrieve or the page itself that you want to retrieve. Right click over the link or the page and select "copy curl/wget for link" or "copy curl/wget for page".
4. Go to the command line and paste

And you shall get the content you asked for, via command line.

Note that you can run that command on ANY computer, not just the computer you were running Firefox on. The cookies and session ids are normally temporal, so they will be valid for a while only.

Note as well that with curl you can use wild-cards. Example:    http://myweb.com/img[0-23].jpg

Easy generation of configuration file for Pools with multiple network labels

One of the new features of VMware Horizon View 5.2 allows you to create pools with different portgroups or network labels.

As part of the configuration process you have to generate a configuration file, and for that you need to run a command with lots of parameters.

I found a way to generate most of the difficult parameters, so you don't spend half an hour (or more) trying different combinations of parameters.

All the info below.


All you have to do is to copy & paste the function below in a "View PowerCLI" console (not the same as "PowerCLI" alone) and then run the command GenerateNetworkConfFile

I include some execution examples below.

If you have issues or suggestions please submit a comment.


#====== SOURCE CODE ============================================================================

function GenerateNetworkConfFile {

# Copyright: Ruben Miguelez Garcia @ 2013 -> http://vmutils.blogspot.com

# Log in to VC if not already done
echo ""
$loggedinvc = read-host -Prompt "Have you already logged into the vCenter used by View using the command Connect-VIServer ?  [y]/n"
if ($loggedinvc -eq "n"){ Connect-VIServer };

# Request a few details
echo ""
$vmname= read-host -Prompt "What is the name of the VM you want to generate a network configuration file for?"


echo "";
$MaxVMsPerNetworkLabel= read-host -Prompt "How many IPs do you want to set up as default limit per network label? [240]"
if ($MaxVMsPerNetworkLabel -eq "") { $MaxVMsPerNetworkLabel="240"; }


# The two functions below come from the View PowerCLI documentation (Integration guide)
# The 1st function has a small modification
#------------------------------------------
function Get-SnapshotPath {
 param(
  $Snapshot
 )
 PROCESS {
  if ($Snapshot) {
   $SnapPath = $Snapshot.Name
   do {
    $SnapPath = "$($snapshot.Parent.Name)/" + $SnapPath
    $Snapshot = $Snapshot.Parentsnapshot
   } while ($Snapshot.Parent -ne $null)

   #$SnapPath = "/" + $SnapPath
   if ($SnapPath[0] -ne "/") { $SnapPath = "/" + $SnapPath }
   $SnapPath
  } Else {
   Write-Error "Snapshot not found"
  }
 }
}
#------------------------------------------
function VVGetPath($InvObject){
    if($InvObject){
 
        $objectType = $InvObject.GetType().Name
        $objectBaseType = $InvObject.GetType().BaseType.Name
        if($objectType.Contains("DatastoreImpl")){
            Write-Error "Use the VVGetDataStorePath function to determine datastore paths."
            break
        }
        if(-not ($objectBaseType.Contains("InventoryItemImpl") -or $objectBaseType.Contains("FolderImpl") -or $objectBaseType.Contains("DatacenterImpl") -or $objectBaseType.Contains("VMHostImpl") ) ){
            Write-Error ("The provided object is not an expected vSphere object type. Object type is " + $objectType)
            break
        }
 
        $path = ""
        # Recursively move up through the inventory hierarchy by parent or folder.
        if($InvObject.ParentId){
            $path = VVGetPath(Get-Inventory -Id $InvObject.ParentId)
        } elseif ($InvObject.FolderId){
            $path = VVGetPath(Get-Folder -Id $InvObject.FolderId)
        }
 
        # Build the path, omitting the "Datacenters" folder at the root.
        if(-not $InvObject.isChildTypeDatacenter){ # Add object to the path.
            $path = $path + "/" + $InvObject.Name
        }
        $path
    }
}
#------------------------------------------


# Get the rest of the details needed automatically
$vm=get-vm -name $vmname
$vc_id=(Get-ViewVC).vc_id ;
$ClusterPath =   "/" + (Get-Datacenter -vm $vm ).name + "/host/" + (get-cluster -vm $vm)
$ParentVmPath = vvgetpath($vm) ;

# Use [0] to get the name of 1st snapshot, [1] for the 2nd, and so on. [-1] for the last one.
# This may not work well if there are multiple snapshot branches.
# The if/else is there to account for cases where there is 1 or multiple snapshots
# If only one, we take that, if multiple, we take the last one
$snaps=$vm | get-snapshot
if ($snaps.gettype().tostring() -eq "VMware.VimAutomation.ViCore.Impl.V1.VM.SnapshotImpl")
 { $ParentSnapshotPath= Get-SnapshotPath ($snaps) }
 else { $ParentSnapshotPath= Get-SnapshotPath ($snaps)[-1] }



# The file will be placed on C:\
$NetworkLabelConfigFile= "C:\" + $vm.name + "NetSpec.txt" ;

# Display what we got  and verify they have valid values
echo ""
echo "Here are the details collected:"
echo "vc_id = $vc_id" ; # Example value: e53c44c5-d5a6-43ec-bab4-f91f7f3fa91e
echo "ClusterPath = $ClusterPath" ; # Example value: /Training/host/ViewCluster
echo "ParentVmPath=$ParentVmPath "; # Example value: /Training/vm/Windows8
echo "ParentSnapshotPath=$ParentSnapshotPath "; # Example value: /Base1/With License/With perf improv/end
echo "NetworkLabelConfigFile=$NetworkLabelConfigFile "; # Example value: C:\Windows8NetSpec.txt


# Generate the configuration file
Export-NetworkLabelSpecForLinkedClone `
-Vc_id $vc_id `
-ClusterPath $ClusterPath `
-ParentVmPath $ParentVmPath `
-ParentSnapshotPath $ParentSnapshotPath `
-MaxVMsPerNetworkLabel $MaxVMsPerNetworkLabel `
-NetworkLabelConfigFile $NetworkLabelConfigFile

}
# End of GenerateNetworkConfFile


#====== USAGE EXAMPLES ============================================================================



######### Using default values in 2 questions

> GenerateNetworkConfFile

Have you already logged into the vCenter used by View using the command Connect-VIServer ?  [y]/n:

What is the name of the VM you want to generate a network configuration file for?: Windows8_empty


How many IPs do you want to set up as default limit per network label? [240]:


Here are the details collected:
vc_id = e53c44c5-d5a6-43ec-bab4-f91f7f3fa91e
ClusterPath = /Training/host/ViewCluster
ParentVmPath=/Training/vm/Windows8_empty
ParentSnapshotPath=/Base Line
NetworkLabelConfigFile=C:\Windows8_emptyNetSpec.txt




######## With problem because VM has no snapshots


PS C:\Users\training> GenerateNetworkConfFile

Have you already logged into the vCenter used by View using the command Connect-VIServer ?  [y]/n: n

cmdlet Connect-VIServer at command pipeline position 1
Supply values for the following parameters:
Server[0]: vc.company.com
Server[1]:

Name                           Port  User
----                           ----  ----
vc.company.com           443   admin

What is the name of the VM you want to generate a network configuration file for?: VM_HW_v9


How many IPs do you want to set up as default limit per network label? [240]:

You cannot call a method on a null-valued expression.
At line:74 char:19
+ if ($snaps.gettype <<<< ().tostring() -eq "VMware.VimAutomation.ViCore.Impl.V1.VM.SnapshotImpl")
    + CategoryInfo          : InvalidOperation: (gettype:String) [], RuntimeException
    + FullyQualifiedErrorId : InvokeMethodOnNull


Here are the details collected:
vc_id = e53c44c5-d5a6-43ec-bab4-f91f7f3fa91e
ClusterPath = /Training/host/ViewCluster
ParentVmPath=/Training/vm/VM_HW_v9
ParentSnapshotPath=/snap1/snap2
NetworkLabelConfigFile=C:\VM_HW_v9NetSpec.txt
Export-NetworkLabelSpecForLinkedClone : PowershellService::ExportNetworkLabelSpecForLinkedClone FAILED, error=No NIC found f
or base VM: /Training/vm/VM_HW_v9 and snapshot: /snap1/snap2
At line:88 char:38
+ Export-NetworkLabelSpecForLinkedClone <<<<  `
    + CategoryInfo          : InvalidResult: (vmware.view.pow...cForLinkedClone:ExportNetworkLabelSpecForLinkedClone) [Expo
   rt-NetworkLabelSpecForLinkedClone], Exception
    + FullyQualifiedErrorId : PowershellService::ExportNetworkLabelSpecForLinkedClone FAILED,vmware.view.powershell.cmdlets
   .ExportNetworkLabelSpecForLinkedClone



############ Another example

> GenerateNetworkConfFile

Have you already logged into the vCenter used by View using the command Connect-VIServer ?  [y]/n:

What is the name of the VM you want to generate a network configuration file for?: Windows7


How many IPs do you want to set up as default limit per network label? [240]: 999


Here are the details collected:
vc_id = e53c44c5-d5a6-43ec-bab4-f91f7f3fa91e
ClusterPath = /Training/host/ViewCluster
ParentVmPath=/Training/vm/Windows7
ParentSnapshotPath=/Base Line
NetworkLabelConfigFile=C:\Windows7NetSpec.txt




################ Checking the output file from console


> cat C:\XP_tiny_emptyNetSpec.txt
#Network Label Configuration Spec

#WARNING! Setting enabled flag to false will
#turn off the automatic network label assignment
#for newly provisioned desktops.
enabled=true

#Parameter Definition for NIC
nic1=Network adapter 1

#Parameter Definition for Network
network01=VDI_Net1
network02=VDI_Net2
network03=VDI_Net3
network04=VM Network
network05=VM Network 2

#Network Label Attribute Definition
#Expected format:
#..maxvm=

####nic1.network01.maxvm=240
####nic1.network02.maxvm=240
####nic1.network03.maxvm=240
####nic1.network04.maxvm=240
####nic1.network05.maxvm=240

Using gatewayInsertExtract PowerCLI script

Running VMware PowerCLI scripts with right click and "Run with Powershell"

Running VMware PowerCLI scripts with right click and "Run with Powershell"


 

------ PowerCLI environment preparation (optional and only needed to set once in the computer)

If you haven't set this before, set it now to avoid multiple warnings

 

Open a PowerCLI console and enter:

Set-ExecutionPolicy bypass

Set-PowerCLIConfiguration -InvalidCertificateAction Ignore

------ end of preparation

 

 

-------- Modification in your script

And now, the first command on your file.ps1 should be this
 
Add-pssnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue
 
 
This will load the PowerCLI stuff so what started as a Powershell script runs now as PowerCLI

------- end of modification
 

If you have problems running the script, open a PowerShell console (not PowerCLI) and run your script from there as ".\file.ps1"
Once you remove all errors, try again to run it with right click.


And here you have two examples that I use to do some repros/testing with VSA 1.0


--------- repro-add-hosts.ps1 -------------
 
# Copyright @ Ruben Miguelez Garcia
 
# Load PowerCLI
add-PSSnapIn  -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue;
 
# Connect to vCenter
Connect-VIServer -server vc -user administrator -password secret
 
# Get folder and Datacenter if exists
$folder=get-folder -norecursion
 
$d=""
$d=Get-Datacenter -Name DC -ErrorAction SilentlyContinue;
if (-not $?){$d = new-datacenter -Location $folder -Name DC ; }
 
# Add hosts
Add-VMHost 10.10.10.10 -location $d -user root -password vmware123 -force
Add-VMHost 10.10.10.20 -location $d -user root -password vmware123 -force
Add-VMHost 10.10.10.30 -location $d -user root -password vmware123 -force
 
 
--------- repro-remove-hosts.ps1 -------------
 
# Copyright @ Ruben Miguelez Garcia
 
# Load PowerCLI
add-PSSnapIn  -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue;
 
# Connect to vCenter
Connect-VIServer -server vc -user administrator -password secret
 
# Remove hosts
Remove-VMHost -VMHost 10.10.10.10 -Confirm:$false
Remove-VMHost -VMHost 10.10.10.20 -Confirm:$false
Remove-VMHost -VMHost 10.10.10.30 -Confirm:$false
 
-----------------------------------------------

New version of pseudo Lab Manager scripts (aka NLclonePodsofVapp.ps1)

[UPDATE NOTE: The work creating, providing and supporting scripts for ESX/ESXi/vCenter has been discontinued. Apologies for the inconvenience.]

I have done some improvements on usability and features based on my needs.

To see a video of the version 3 click here. Click here to read more information about these scripts.



CHANGELOG

------ version 4

• Promiscuous Mode at the vSwitch level is now enabled automatically (this is needed if you have any ESX/ESXi in a VM).
• The vSwitch has now 1016 Ports by default as opposed to the standard number which is 56.
• The limitation of <15 characters for the common name for the Pods has been changed to <23 characters.

------ version 5

• Program detects if VMs in source vApp are not in the same host (this is a requirement).
• Ability to define the cluster as workspace.
• Affinity rules are created for every Pod if DRS is enabled in the cluster (it it exists), independently of whether you select one/multiple hosts or a cluster.

EXAMPLES:

You can define the hosts it will work on manually

.\NLclonePodsofVapp.ps1 -source_vapp SRM-Parent-Pod -cloneName SRM-Test -endN 1 -startN 1 -hosts_list ("10.10.10.20","10.10.10.30")

You can define the hosts it will work on using the cluster the vApp is in

.\NLclonePodsofVapp.ps1 -source_vapp SRM-Parent-Pod -cloneName SRM-Test -endN 2 -startN 2 -use_cluster $true

Or you can just use the host the vApp is in (no need to specify anything)

.\NLclonePodsofVapp.ps1 -source_vapp SRM-Parent-Pod -cloneName SRM-Test -endN 3 -startN 3

You can create N Pods in these two ways (same result)

.\NLclonePodsofVapp.ps1 -source_vapp View-Parent-Pod -cloneName View-Test -endN 8

.\NLclonePodsofVapp.ps1 -source_vapp View-Parent-Pod -cloneName View-Test 8

If later on you want more Pods with the same root name you have to specify the starting point. This creates Pod09, Pod10,..., Pod15

.\NLclonePodsofVapp.ps1 -source_vapp View-Parent-Pod -cloneName View-Test -endN 15 -startN 9

If start and end is the same number, only one Pod with that number is created

.\NLclonePodsofVapp.ps1 -source_vapp VSA-Parent-Pod -cloneName VSA-Test -endN 7 -startN 7

The affinity rules are created automatically whenever it is possible. You don't need to request it.

This means now that if DRS is enabled, you can create multiple Pods that at power on will be distributed across the hosts by DRS and the affinity rules will keep the VMs in each Pod in the same host.



As VMware vCenter Lab Manager will not see further major releases, users have no option but to move to vCloud Director. But as Mike clearly says, vCD is not designed to be a replacement for Lab Manager. For those who see vCD as 'too big for what I need' this scripted solution comes handy.

Using pseudo Lab Manager

An example of how in a few minutes you can have this script cloning vApps for testing/teaching/etc.

I suggest you watch it in fullscreen or large size.

Scripted linked-cloning & pseudo-LabManager with PowerCLI

[UPDATE NOTE: The work creating, providing and supporting scripts for ESX/ESXi/vCenter has been discontinued. Apologies for the inconvenience.]


I have created a group of scripts that allow you to:

• Create a linked-clone of a VM
• Create a linked-clone of a vApp (creates a vApp and then puts inside linked-clones of the VMs on the original vApp. This new vApp does NOT inherit the vApp configuration from the parent)
• Create N linked-clone vApps, putting the VMs inside each vApp in an isolated network (PortGroup).

Basic Linked clones:

One of the many features of VMware View is the automated creation of pools of Desktops that share the same parent VM (linked clones).

I have created a PowerCLI script that allows you to create linked clones from a Parent VM in a matter of seconds. These VMs are pure linked clones, so the script does not perform any reconfiguration tasks on the OS.

Each one of those linked-clones know that it is a linked-clone, and therefore, when you delete it from disk it will not delete the virtual disk(s) of the parent VM.

I must say that I reused the code of Keshav Attrey [0] for the basic linked clone operation.

Pseudo LabManager Pod cloning + Isolation:

VMware LabManager allows you to clone multiple times virtual environments (groups of VMs in a can, isolated from other VMs from the network perspective, also know as Pod of VMs). And the best of all, is that those clones are linked-clones, which reduces drastically the time and space needed for their creation. For more information see its features [1].


Some of the benefits [2] of Vmware Lab Manager are,

- Reproduce bugs and reduce time spent in the debug phase

- Better management of joint resource across teams


Indirect benefits are,

- Delivering better product support

- Easier troubleshooting for customer production problems

- Improved productivity and efficiency

- Reduce time finding spare servers

- No need to hoard servers and storage

- Save power, space & HVAC


The technical benefits are,

- Provision systems quickly.

- Restore previous configurations.

- Quickly make changes to a configuration, possibly via user self-service.

- Recycle system resources for other uses.



However LabManager has some requirements like a LabManager server and a Database. Those and other requirements may not allow you to use LabManager or it may not be the best idea if you want a simple & quick cloning of a Pod. Actually you can put the VMs of the group in a vApp and clone it, but it will take quite some time and storage consumption even if you are using thin provision disks.


For that reason I have created another PowerCLI script that does the same core 'magic' that LabManager does but with very few requirements, and without LabManager expertise needed. Just one single PowerCLI script to create the Pods and another to delete them (you can also delete them manually). The resulting VMs are more flexible than LabManager Pods because you can directly from vSphere Client add/change/remove network/CDs/vDisks/etc while they are running. And they are thin-provisioned whenever you want, so you don't have to do some extra magic [4] to convert them to thin.


This other script does the following things:

• Creates an internal only vSwitch
• Creates a Port group on that vSwitch for every Pod
• Creates the N linked clones of the specified vApp

LabManager had its own user interface to access the Pods. In this case you use the vSphere Client, which allows multiple concurrent access/users. And because each Pod is a vApp, you can modify the permissions/rights on each one. For example, a user may be able to log in into vCenter, but he/she may only be able to use the vApp/Pod he/she has been assigned to. In other words, you stay in control, while you get the benefits and simplicity of this solution.

========= Commands help (using PowerShell help) =========

Just type:
# help .\commandname   (if you are in the folder of the program)

Every script has documentation and examples.

======== General Instructions ================

1. Install PowerCLI (in vCenter or a computer able to reach vCenter)
http://www.vmware.com/support/developer/PowerCLI/index.html

2. Open PowerCLI
Start > Programs > VMware > VMware vSphere PowerCLI > VMware vSphere PowerCLI.

3. Connect to vCenter with:
# connect-viserver -server
or
# connect-viserver -server localhost  (if you installed the PowerCLI on the vCenter)

4. By default, Microsoft has prevented the running of custom PowerShell scripts. Run this to change that behavior.
# Set-executionpolicy unrestricted
The change/setting will stay even after reboot, so don't need to set again.

5. Copy the scripts to some location and move to it from PowerCLI/PowerShell

6. In vCenter
Put some VMs in a vApp. Minimum two.
Take a snapshot of each one of those VMs. Ideally they would be powered off, but it is possible to have them running (don't snapshot the memory).
Make sure they are all in the same host.

7. Run (example):
# .\NLclonePodsofVapp.ps1  SourceVappName  Student  3

This will create Student-Pod01, Student-Pod02, Student-Pod03 and a vSwitch called vSwitch-Internal-Student

8. You can power on the Pods/vApps and verify that they are linked clones of the original VM, and that the VMs inside each Pod are in an isolated group.

9. When you are done with your experiments/tests and want to destroy the Pods and the vSwitch, run:
# .\removeLclonePods.ps1 Student
This will remove
vApps: Student-Pod01, Student-Pod02, Student-Pod03
vSwitch: vSwitch-Internal-Student

The parent vApp/VMs won't be touched, as these linked-clones are aware of their nature and therefore won't delete the disks of the parent VM when they are removed from disk.




======= Requirements =========
Note that these scripts have gone through limited testing. For certain environments more requirements may apply. If you find any not listed here please let me know.

• PowerCLI

• The parent VMs must have an snapshot taken prior to the cloning operation

• All the VMs in the vApp must be in the same host. It makes no sense to have the VMs forming a vApp/Pod spread across hosts because the vSwitch they are connected to has no physical NIC, so they must be together. The script NLclonePodsofVapp.ps1 allows to specify multiple hosts for the creation of the Pods, but you still need to ensure they are together. Specifying multiple hosts only ensures that the networking needed is created in all the hosts provided, but does not move the vApps/VMs.

• The VMs inside the vApp must have unique names in vCenter inventory, even across Datacenters. If there are duplicated names, you will get an error like:

          Method invocation failed because [System.Object[]] doesn't contain a method named 'CloneVM_Task'

• The size of the common name for the Pods has to be <15 characters.

======== FAQ =================

Q: Do I need vCenter to use this?
A: Yes. ESX/ESXi doesn't know anything about cloning a VM or about what a vApp is.

Q: Which version of PowerCLI should I use?
A: Latest. 4.x or higher.

Q: Will this work with ESXi 5.0 / PowerCLI 5.0?
A:Yes.


Q: Can I make a linked-clone of a linked-clone?
A: Yes you can. You just need to ensure it has a snapshot before you clone it.

With VMware and these scripts you can do incredible deployments with very very little space.

See here an example of an complete virtualized SRM deployment (SRM in a box):

Parent-W2008
    -> L_with_vSphereClient (Installed vSphere Client on the windows and took a snapshot)
        -> L_VC_Protected (Changed hostname/IP and installed vCenter Server)
        -> L_VC_Recovery  (Changed hostname/IP and installed vCenter Server)
    -> L_SRM_Protected (Changed hostname/IP and installed SRM Server)
    -> L_SRM_Recovery  (Changed hostname/IP and installed SRM Server)
Parent-vESX
    -> L_vESX_Protected (Reset Defaults on DCUI (Direct Console User Interface) and configured networking)
    -> L_vESX_Recovery  (Reset Defaults on DCUI (Direct Console User Interface) and configured networking)
Parent-StorageSimulator
    -> L_Storage_Protected (Reconfigure networking)
    -> L_Storage_Recovery  (Reconfigure networking)

Parent = Real/Full VM
L_ = Linked clone

So here you have 8 VMs created from only 3 real/full VMs. And 2 of those 8 are linked-clones of another linked-clone.

Once all these L-VMs have a snapshot, you can put them all in a vApp and create N SRM boxes in a box using the script NLclonePodsofVapp.ps1. Minimum space consumption with  maximum flexibility. They will need quite some memory, but the memory sharing will be very high. I have been able to run 12 VMs of 4GB RAM each one in a physical ESXi with only 16GB RAM and I wasn't using it all yet.

======== Notes ===============

• If you are going to have VMs running ESX/ESXi, you NEED to enable Promiscuous mode on the vSwitch they are connected to.

• If you will connect any of the linked clones to the external network in a way that they face a clone of itself, ensure the NIC(s) of the Parent VM is set to auto, otherwise there will be a MAC conflict once they see each other. That Auto setting generates new MACs on the NICs of the clones of that VM.

======== Source code  ===========
http://communities.vmware.com/thread/324193

======== References ===========

[0] Creating a linked clone from a snapshot point in PowerCLI : http://www.vmdev.info/?p=40

[1] VMware vCenter Lab Manager : http://www.vmware.com/products/labmanager/features.html

[2] Why Developers & Testers will LOVE Vmware's Lab Manager : http://geekswithblogs.net/SabotsShell/archive/2008/07/13/why-developers-amp-testers-will-love-vmwares-lab-manager.aspx

[3] VMware vSphere PowerCLI : http://www.vmware.com/support/developer/PowerCLI/index.html

[4] Automatic thinning of virtual disks with makeThin : http://vmutils.blogspot.com/2011/06/automatic-thinning-of-virtual-disks.html

======= Versions =============

When I saw the Post on [0] I barely knew how to use PowerShell/PowerCLI. It took me 1.5 days to get this script (v1) together. The part of the vApps came later.

# v1 All basic functionality working perfectly. The clones are placed inside vApps.

# v2 Added ability to link-clone a vApp so it is not needed to reference the list of VMs individually.

# v3 Added optional posibility of giving a list of hosts on which the networking infrastructure will be created.

======= keywords =============
LabManager, Lab Manager, replacement, alternative, vCD, vCloud Director, less complexity

Screenshots (of the version 1)

Automatic thinning of virtual disks with makeThin

Here I present my latest tool for ESX. An script that automatically converts thick virtual disks to thin.

It has been used for almost a year in my team across different centers and the storage savings are >50%.


makeThin Documentation

Happy thinning!

[Update 8/January/2013]
Victor was kind enough to create a fork of this program for ESXi. You can find all the info on https://github.com/terreActive/makeThin
[End of update]


PS: If you link the documentation/script, please link to this post or the blog. The location of the document on the back may change, the blog will not.

Possibly, the end of the snapshots story.

The way 'Delete all' snapshots does the job has changed with some recent patches.


Which patches change this feature?

• VMware ESX 3.5, Patch ESX350-201006401-SG: Updates VMkernel, VMware Tools, hostd, VMX, VMnix (http://kb.vmware.com/kb/1022899)

• VMware ESXi 3.5, Patch ESXe350-201006401-I-SG: Updates firmware (http://kb.vmware.com/kb/1020052)

• VMware ESX 4.0, Patch ESX400-201006201-UG: Updates the VMware ESX 4.0 Core and CIM components (http://kb.vmware.com/kb/1017721)

• VMware ESXi 4.0, Patch ESXi400-201006201-UG: Updates Firmware (http://kb.vmware.com/kb/1017739)

What are the changes?

Before these patches, commiting more than one snapshot using the Delete all option from the Snapshot manager would require additional space to perform the operation. The amount of extra space required was directly related with the amount of snapshots and their size.

The patches modify the Delete all snapshots operation to commit every snapshot of the chain directly to the Base Disk(s) of the virtual machine. With this new algorithm:

• If the Base Disk is preallocated (thick provision), no extra space will be required for the Delete all operation. The Base Disk will not grow as it is preallocated or thick.

• If the Base Disk is non-preallocated (thin provision), the base disk will grow only on committing information from the snapshots. Each thin provision disk may grow up to its maximum size as mentioned in the Provisioned Size option in the Virtual Machine settings for the disk.

A quick way to know how much the Virtual Machine disk utilization can grow (whether you are using thin or mixed disk types) is taking a look to the Summary tab of the Virtual Machine.
On the right panel you will see "Provisioned Storage" and "Used Storage". The difference is how much the Base Disks can grow.

However, if the VM has disks in different Datastores, this amount will be shared between them.




Can I change the type of the virtual Disks when it has snapshots?
You cannot change the type of a Virtual disk with snapshots directly. You can perform one of these oprerations to change the type of disks:

• When you do Storage Vmotion (SVmotion) you can change the type of disks, but not having snapshots is a requirement even for SVmotion.

• When you Clone the virtual machine you can change the type of disks on destination virtual machine, however it will be the same type for all the disks.

• If you clone from Service Console, you can select any type of disk for the destination, however you can not clone from the Service Console if the virtual machine is running.

What happens if I click 'Delete all' while the virtual machine is running?
If the virtual machine is running when you click Delete all from the interface or run vmware-cmd removesnapshots an additional snapshot is created to accommodate the incoming I/O while all the other snapshots get committed to the Base Disk. The size of the snapshot can grow depending on the I/O activity. It is ideal to reduce the I/O activity to facilitate the process.

That last snapshot will commit to the Base Disk at the very end of the process, after this all the snapshot files will be deleted.



Will each snapshot file get deleted as soon as it gets committed to the base disk releasing space on the Datastore?

No, all files are deleted together after completing the process.



These patches will invalidate the space needed given by SnapVMX and will make the snapshots troubleshooting much easier (I'll update them when I have some time). They are the result of a personal effort that has finally reached the end. After creating the snapshots troubleshooting guide and script, I decided that was time to work on eliminating the problem rather than just resolving it.

Well, the patches are out there. My advice, install them. I have been waiting long time to be able to announce them.



[Update June/2011] These patches were the result of a personal battle to eliminate the problem from its root. Initially I created a troubleshooting procedure and a helper script (SnapVMX) to minimize the time fixing the consequences of the problem. After a while, I changed my mind and focused on eliminating the problem itself. With the time we have seen it was the right thing to do. The daily long calls about problems originated by snapshots are now history.

Preventing surprises with VMs running on snapshots.

It's good to know how to resolve the problem. But it's better to prevent it.

Configuring VMware vCenter Server to send an alarm when virtual machines are running from snapshots



No console scripts, no powershell, no hassle, just use one of the for many unused and unknown features of vSphere, alarms.

Troubleshooting Virtual Machine snapshot problems and SnapVMX

These two documents may help you to understand and resolve the problems of VMs with snapshots.

-- Troubleshooting Virtual Machine snapshot problems
-- SnapVMX Documentation

Feedback always welcome.

If you have a problem with snapshots and need support, please call VMware support.

If you have questions about something unrelated with the topic presented here (VDR, PowerCLI, Workstation, ...) I suggest you find the appropiate forum in the VMware communities and post it there.


If you link the script/troubleshooting guide, please link to this post or the blog. The server hosting the docs may change, the blog will not.