vCenter 5 refuses to start because duplicate datastore

Please take note that you must do this only with proper guidance from VMware Support. What I did below is not supported and is not recommended.

Plus, I got my vCenter DB corrupted as a result along the way, and I had to spend more time than I thought I would. So you have been warned.

 

vCenter service will not start. The error at vpxd is duplicate name for the datastore named Templates. I’ve manually unmounted the Templates datastore from each ESXi, but still got the same error. So my guess is vCenter database is somehow corrupted.

Below is the log.

------------------

2012-04-28T18:16:26.458+08:00 [02520 info 'Default'] [Vpxd::ServerApp::Init:898] Calling: VpxdMoHost::Init()

2012-04-28T18:16:26.458+08:00 [02520 info 'Default'] [Vpxd::ServerApp::Init:899] Calling: VpxdDatastore::Init(gDB)

2012-04-28T18:16:26.458+08:00 [02520 error 'Default'] Duplicate name (Templates) in datastore folder

2012-04-28T18:16:26.458+08:00 [02520 error 'Default'] [Vpxd::ServerApp::Init] Init failed: VpxdDatastore::Init(gDB)

2012-04-28T18:16:26.458+08:00 [02520 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 2078 ms

2012-04-28T18:16:26.474+08:00 [02520 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down...

2012-04-28T18:16:26.474+08:00 [02520 info 'Default'] Wrote uptime information

2012-04-28T18:16:36.381+08:00 [02536 warning 'VpxProfiler' opID=SWI-46e11f30] VpxUtil_InvokeWithOpId [TotalTime] took 12000 ms

2012-04-28T18:16:48.382+08:00 [02536 warning 'VpxProfiler' opID=SWI-7871f574] VpxUtil_InvokeWithOpId [TotalTime] took 12000 ms

2012-04-28T18:17:00.383+08:00 [02536 warning 'VpxProfiler' opID=SWI-ef0a9776] VpxUtil_InvokeWithOpId [TotalTime] took 12000 ms

2012-04-28T18:17:12.376+08:00 [02536 warning 'VpxProfiler' opID=SWI-ff16e089] VpxUtil_InvokeWithOpId [TotalTime] took 12000 ms

2012-04-28T18:17:24.358+08:00 [02536 warning 'VpxProfiler' opID=SWI-dccf3641] VpxUtil_InvokeWithOpId [TotalTime] took 12000 ms

2012-04-28T18:17:24.702+08:00 [02520 info 'Default'] Forcing shutdown of VMware VirtualCenter now

 

With the help of some kind experts, I connected to the database.

Below are some steps I used to recover. Please note that the actual recovery experience was not as smooth as this one. I’m just documenting the key steps here.

 

 

 

 

So I go to the DS_Assignment table and delete them first.

 

A warning as it is not undoable.

 

The deletion went successfully.

So I went back to the VPX_Datastore table and deleted the duplicate entry.

And my VC is up J

 

 

 

How to setup an integrated software token for View iPad client

If you setup the embedded RSA software token, you only need to enter the PIN. The embedded token will generate the passcode for you. So you no longer need to copy paste. This increases usability from user point of view, and make the login process smoother.

 

First, you need to download from EMC RSA, the TokenConverter program. You can download it freely from EMC, but you need to register first.

Here is the download page: http://www.emc.com/security/rsa-securid/rsa-securid-software-authenticators/iphone-and-ipad.htm

From there, click on the link: http://rsa.com/poc2/rsa-form-holder-swdl-eula.htm?asset_title=Web_Toolbar_Token_Converter_Ongoing&asset_id=sw_download_eula&asset_url=http://www.rsa.com/Web_Toolbar_Token_Converter_Ongoing

After registration, here is the software you need to download.

 

In the example below, I’ve downloaded and unzip the software into a directory. I also saved my token file (*.sdtid file), given my IT department, in the same directory.

 

The command to execute is below. The tokenfile.txt is the output file, you can name it whatever you want. It will only consist of the “URL”, which you then copy to your iPad.

I specified –p 8029 as that IT sets a password to my token file.

 

The output file tokenfile.txt looks something like this: com.rsa.securid://ctf?ctfData=200012288888861073511117714535255644265676670025047324406172127616014406111174444

Don’t copy the above, as I’ve changed the numbers J

 

Launch your View client in iPad.

In the External Token, fill the 3 fields below.

Token Description is just plain text. I called mine “iwan”. This is only useful if you have multiple token.

In the URL, paste that long ctf string that you get in your output file.

If your IT specified a password when giving you the token file, you must key in here. So in my case, I keyed in 8029 again.

 

Here is what it looks like after I installed it. I

 

And here is the result as a user. Notice I just need to key in the PIN. The code is automatically generated. So I no longer have to do the following:

1.      Press the home button in iPad.

2.      Find the RSA apps, and manually launch the RSA app

3.      Key in the PIN in RSA

4.      Copy the code

5.      Switch to the View client app.

6.      Paste the code

 

So 5 steps were eliminated and it becomes an integrated process. No more dealing with 2 apps J

 

 

How to setup Initiators in NetApp

As Initiator is related LUN, you set it up under the LUNs tree. Choose the tab Initiator Groups.

Click the Create button.

This brings up the Create Initiator Group dialog box. In this box, I’ve specified the name, and choose the OS.

Notice you must choose the type, as naturally the initiator value is different.

 

And in this tab you specify the members. I’ve only 1 ESX so I’ve added that below.

 

Once the initiator group is created, you can use it to map to the LUN.

So select a LUN, then click edit to add the initiator group. In this case, I’ve precreated 2 groups, so both appears here.

Be careful in setting the LUN ID. It must be unique, so plan in advance if you have many LUNs and many ESXi

 

That’s basically it. Do a rescan and it will appear on the ESX. Below is what I got after I did the rescan.

 

 

How to create LUN in NetApp

To create a LUN in NetApp, select LUNs from the tree to bring up the LUN screen.

Click on Create button, as shown on the screen below.  The LUN Wizard dialog box will appear.

Click Next.

 

The next screen is to type the LUN name and size. Make sure you choose VMware from the Type drop down field, as the default is not VMware. I chose thin provisioned here. Not sure what the impact as the Volume is already thin provisioned and dedupe. Notice dedupe is not done at LUN level, but volume level.

 

A LUN must be placed in a volume. Since I’ve created the volume earlier, I’d choose an existing volume. Clicking the browse button will bring up the volume tree.

 

The next screen is to map the LUN to ESX (initiators, specifically). I will cover the initiator creation on separate entry.

 

That’s basically it. This is the summary screen before it begins the creation.

 

And here is the result. It takes a few seconds to create the LUN.

 

That’s it.

 

How to create a new aggregate in NetApp

Here is the steps to create a new aggregate in NetApp. You need to create at least 1 aggregate as you should not use the aggr0 as that’s where the root volume is.

First, go to the Aggregates list. From there, click the Create button.

The dialog box will appear, starting the wizard.

 

Give it a name. Since I’m only going to have 1 aggregate that combines all the volumes, I’ve given it the name below. I think using different aggregate will result in a waste (capacity and IOPS) as 1 physical spindle can not be shared among aggregate. So if Aggregate1 has 200 spindles, and Aggregate2 has 20 spindles, and Aggregate2 cannot cope with the IOPS, it cannot use the spindles in Aggregate1.

 

I only have 1 disk group, which was created earlier.

 

Clicking the Next button give me the screen below. I need to choose the disks, so I will click Select Disks.

 

From here, I chose 20 disks. I’m setting aside 1 disk as spare. It is automatically chosen. I don’t have to specify which one is spare.

 

From the 20 spindles, I chose to create RAID (8+2), so I specified 10. It automatically creates the 2 RAID group.

 

This is the last screen.

 

The process below takes only a few minutes. In fact, I think it’s less than 1 minute as it’s pretty fast!

 

This is the result.

 

And this is the result. Notice it even tells me which physical spindles it uses for parity.

 

And here is the volumes subtab looks like. I’ve created 6 volumes here.

 

Hope that helps.

 

 

How to create a volume on NetApp

From the Volumes list, click on the Create button.

The “Create Volume” dialog box will pop up, as shown below.

The name cannot include space or other special characters, except underscore.

You much choose the protocol type at this level. So this requires some planning.

Normally, I’d dedicate 20% of space for snapshot. I like this feature that the usable disk space is automatically reduced, as that means I don’t have to worry about the snapshot taking up space beyond the allocated volume.

 

 

The “Storage Efficiency” tab lets you do the dedupe and compression. Since my case is a demo lab, so I’ve turned on all features. Yes, the lab is live, but then that’s how we learn right J

 

 

How to delete a volume in NetApp

Here is the step to delete volume using the NetApp System Manager.

 

From the Volumes list, click the Volume you need to delete.

Click Delete.

 

In the above case, since the volume is offline, you will get a warning. So we need to bring the volume offline first. To do that, choose Status à Offline.

And once you bring it offline, you can delete it.

NetApp: enabling features with System Manager

How do you enable FC or iSCSI features in NetApp with System Manager?

 

I was trying to find out why I could not create a FC Volume on my newly setup NetApp. The only option was NFS. I checked the Protocols section in tree, and it was only showing NFS and CIFS.

To show the other protocols, do this:

1.       Go to the System Tools à Licenses section.

2.       Click on the FC/FCoE line.

3.       Click Enable link at the bar. Wait for a few seconds, and it will be enabled.

4.       Do the same for the iSCSI line.

As you can see from the picture below, I now have FC/FCoE and iSCSI under the Protocols section.

 

Can you storage vMotion the vCenter VM itself?

I got a question from customer on that. Interesting question, and my thought is it should be, as it’s transparent to it. The operation done by the storage vMotion, is independent of the operation used by vCenter to do its management (e.g. monitoring hosts). I don’t think vpxd and vpxa will be affected.

 

Anyway, since I needed to upgrade the lab (NetApp & EMC are upgrading their respective arrays. Thank you William, Ken, Scott!), I had to move a lot of VM. Below is the core infrastructure. The lab is called VMSG, stands for VMware Singapore.

So I did storage vMotion for all the VMs below. It took some time, as I’s migrating multiple VMs (I let them run in parallel, in queue).

 

The vCenter is called VMSG-VC-004, and I moved it from an NFS datastore to the ESXi local datastore.