evan All American 27701 Posts user info edit post |
0:00:01:09.180 cpu4:1048)LVM: 4469: vmhba0:0:0:6 may be snapshot: disabling access. See resignaturing section in SAN config guide.
this is showing up on the service console of one of my hosts... but only this one... so i'm certain the SAN doesn't need to get new IDs...
any ideas? i'm out of them, and google wasn't helpful. 2/8/2008 7:58:33 PM |
El Nachó special helper 16370 Posts user info edit post |
must...resist...urge for parody thread 2/8/2008 8:17:55 PM |
gs7 All American 2354 Posts user info edit post |
Heh, he asked... 2/8/2008 10:56:04 PM |
evan All American 27701 Posts user info edit post |
fixed after 5 hours of work
p.s. hope that this never happens to your cluster 2/9/2008 12:04:09 AM |
abbradsh All American 2418 Posts user info edit post |
late night? 2/9/2008 1:01:12 AM |
evan All American 27701 Posts user info edit post |
yes
home now though 2/9/2008 1:07:58 AM |
gs7 All American 2354 Posts user info edit post |
What solution did you come up with? 2/9/2008 1:25:55 AM |
Scary Larry Suspended 644 Posts user info edit post |
don't be that guy
don't be responsible for the post I find on Google that includes a specific error message with no solution/workaround 2/9/2008 8:39:10 AM |
evan All American 27701 Posts user info edit post |
haha, basically, here's what i did:
1) go into the clariion switch admin to see what hosts had a lock on those LUNs 2) put those hosts into maintenance mode, migrate off VMs, shut down 3) some VMs wouldn't migrate off because there were phantom processes on other hosts (which i think caused the problem in the first place)... had to ps -efwww and grep to find the offending processes and kill them 4) copy all the files off of that LUN to the local datastore 5) recreate the LUN 6) bring hosts back up, take out of maintenance mode 7) have all hosts rescan for datastores to regenerate the scsi/uuid's
now you see why it took 5 hours 2/9/2008 9:28:50 AM |
llama All American 841 Posts user info edit post |
Quote : | "don't be that guy
don't be responsible for the post I find on Google that includes a specific error message with no solution/workaround" |
amen!
not having much experience with esx, what do you mean by "migrate off"?2/9/2008 9:33:54 PM |
evan All American 27701 Posts user info edit post |
vmotion
http://www.vmware.com/products/vi/vc/vmotion.html
that, HA, and DRS are pretty much the coolest fucking things ever
they make my life so much easier, especially since we have 30 ESX hosts (each have 4 quad core opterons (16 logical cores!) and either 32 or 64GB of ram, so each can run around 20-30 VMs, depending on usage...)
HA and DRS make it so if one ESX host fails in our cluster, it notices that, rebuilds the vmx file (all the disk/swap/ram files are stored on the SAN that all the ESX hosts can see), and brings the VM back up on another host... all seamlessly... and it pages me and lets me know what's going on
vmware esx server/virtualcenter is the SHIT. check it out sometime. 2/10/2008 4:04:47 PM |
llama All American 841 Posts user info edit post |
damn, I wish xen did all that
sounds like a combination of xen + RH cluster suite, but I bet its much easier to use 2/10/2008 4:47:50 PM |
evan All American 27701 Posts user info edit post |
we're looking at xen actually
running a small test
the hypervisor works better in xen, but vmware is catching up
vmware is indeed easier to manage, but that's irrelevant, as i'm already pretty good with xen.
plus... xen is cheap... vmware is mega $$$. 2/10/2008 7:18:05 PM |
llama All American 841 Posts user info edit post |
I'm a fan of xen, but I don't know many people that like it
para-virt ftw 2/11/2008 5:26:37 PM |