VMFS Datastore inaccessibility or corruption during server decommissioning
search cancel

VMFS Datastore inaccessibility or corruption during server decommissioning

book

Article ID: 386248

calendar_today

Updated On:

Products

VMware vSphere ESXi

Issue/Introduction

  • One or more servers previously used for ESXi are being decommissioned.
  • After a reboot or rescan of an ESXi host a datastore/datastore become inaccessible.
  • ESXi hosts start logging messages indicating corruption as seen in /var/run/log/vmkernel.log 
WARNING: FS3: 608: VMFS volume DATASTORE_LABEL/DATASTORE_UUID on naa.#################################:1 has been detected corrupted

 

Environment

VMware vSphere ESXi 7.x

VMware vSphere ESXi 8.x

 

Cause

  • sdelete or another similar tool has been employed to overwrite data on disks on the server being decommissioned.
  • Shared vmfs datastores remain mapped to the server being decommissioned, leading to sdelete/other tool overwriting one or more shared vmfs datastores. 

If this has occurred:


1. Primary partition table may be overwritten: 

 

partedUtil getptbl /vmfs/devices/disks/naa.################################

Error: The primary GPT table is corrupt, but the backup appears OK, so that will be used. Fix primary table ? diskPath (/dev/disks/naa.################################) diskSize (16777217280) AlternateLBA (1) LastUsableLBA (16777217246)
gpt
1044333 255 63 16777217280
1 2048 16777215999 AA31E02A400F11DB9590000C2911D1B8 vmfs 0

 

2. hexdump of the datastore shows that the vmfs header (including usually zeroed buffer blocks) is overwritten with random data:

hexdump -C /vmfs/devices/disks/naa.################################ | more

00000000  58 dd f3 79 2b 9e 6a 78  ef 97 16 ae f7 93 d8 9d  |X..y+.jx........|
00000010  a7 ed 33 dc 6c d8 f6 ca  47 d9 24 bc 68 65 e8 95  |..3.l...G.$.he..|
00000020  10 53 07 0c 04 4b ae ee  48 4a d8 8d 31 2e 89 b2  |.S...K..HJ..1...|
00000030  62 a4 81 20 ec be 32 e2  d1 f2 85 29 d1 96 c6 39  |b.. ..2....)...9|
00000040  72 4a 43 e3 7c d2 4b 55  29 18 c2 26 49 28 a2 2a  |rJC.|.KU)..&I(.*|
00000050  15 7f 61 cc 0c ae 8f 9a  27 f9 26 88 b0 a7 04 bb  |..a.....'.&.....|
00000060  ee cf 4c 30 5f e2 bd 0e  3d b5 05 8c 4d d2 36 f2  |..L0_...=...M.6.|
00000070  07 9f 37 b5 9f 21 0d 48  bb 57 02 b5 be b0 b5 47  |..7..!.H.W.....G|
00000080  6c 52 72 1b 3a 62 3f eb  0a 26 92 4c 73 f4 d9 31  |lRr.:b?..&.Ls..1|
00000090  4c 6b cd 61 51 d5 3b ce  42 35 75 a7 0f 83 0c 39  |Lk.aQ.;.B5u....9|
000000a0  f2 a7 64 0a ee df 7c dc  b9 e3 df 0e 5e d2 ae ce  |..d...|.....^...|
000000b0  94 e3 80 90 be 51 4c 50  df 84 09 56 d6 be cc 32  |.....QLP...V...2|
000000c0  4a 3d 54 c7 00 9e 7c ed  ee 4f 26 6f 8d 37 b2 46  |J=T...|..O&o.7.F|
000000d0  ef 6a 8e d0 66 f9 5a 18  15 e1 48 87 c2 a5 d9 24  |.j..f.Z...H....$|
000000e0  47 62 20 37 18 7d 47 95  0b bf bc 82 13 29 c2 93  |Gb 7.}G......)..|
000000f0  ec fa ad 62 08 fa bd 05  48 0f 26 6f 47 f9 1e d3  |...b....H.&oG...|
00000100  61 fe 52 86 52 c4 f2 b4  7e c1 ca fd 3d 03 f1 46  |a.R.R...~...=..F|
00000110  b4 ca 8d 3c 47 e7 5c f9  93 a5 ba d9 10 fa 69 61  |...<G.\.......ia|
00000120  94 ef 3c d2 7d 56 f9 ce  f8 ab 4d 81 2a 0a 8b 30  |..<.}V....M.*..0|
00000130  0c 7f 06 06 65 af d6 84  90 42 34 99 88 5a dc 11  |....e....B4..Z..|
00000140  c8 3f fc f7 b4 7d 0b c3  e3 c5 20 f0 1c 12 4e 03  |.?...}.... ...N.|
00000150  99 47 e9 8b 43 b2 38 90  73 f6 28 2f 80 b7 0a 00  |.G..C.8.s.(/....|
00000160  8b 51 3a 97 ca 3e e3 fc  04 c5 f0 d7 27 a1 35 9e  |.Q:..>......'.5.|
00000170  58 c5 20 91 d4 8e 96 2f  52 d8 fe b3 a6 65 62 2d  |X. ..../R....eb-|
00000180  43 c8 0d bc 0f 74 eb 3e  98 b1 18 0f 8d e4 81 36  |C....t.>.......6|
00000190  80 fa 96 d4 9b 0e ab cd  43 ff af 4f 05 c8 61 e0  |........C..O..a.|
000001a0  cd 77 63 90 0b 0c 7e 38  3f e0 f7 24 f6 3c b8 d8  |.wc...~8?..$.<..|
000001b0  5b de 55 17 b6 38 4b c4  ba 1f d2 c9 f8 ae c3 15  |[.U..8K.........|
000001c0  70 9a ad 7d 73 26 7e 3b  c7 e7 62 cb 01 88 96 49  |p..}s&~;..b....I|
000001d0  48 e6 d9 0b 2c 97 fe 5f  ee 07 eb fd 51 74 46 3e  |H...,.._....QtF>|
000001e0  a7 99 04 5e 72 9d b9 5d  5e 25 72 e0 94 32 99 a6  |...^r..]^%r..2..|
000001f0  04 bf 70 37 d6 77 d7 45  1d be 37 06 a2 3a bd d0  |..p7.w.E..7..:..|
00000200  8f 4d f8 74 4b 2a 96 dd  c6 22 42 eb e8 da e6 24  |.M.tK*..."B....$|
00000210  ca e7 09 e3 3e 7d 9e 1e  1c 5f 50 e9 e4 d6 20 fe  |....>}..._P... .|
00000220  fc 7d 61 23 54 64 4d 01  6b 20 2c d8 80 7f a3 41  |.}a#TdM.k ,....A|
00000230  76 97 6d 2d 46 a4 ed d4  95 bc 54 6c a2 9a aa e0  |v.m-F.....Tl....|
00000240  97 bc 0a 22 97 f0 36 dc  c2 7a 1d 91 2c 3c ef a7  |..."..6..z..,<..|
00000250  0e b0 10 a2 6b 5b fa c5  e3 6c 0c 63 0b c2 2d 91  |....k[...l.c..-.|
00000260  60 c5 ea 3e d7 32 e3 f9  ec b3 c4 bd be f8 b8 79  |`..>.2.........y|
00000270  36 d8 54 87 0b 9d 45 fe  a6 e5 d7 e8 23 80 3b b2  |6.T...E.....#.;.|
00000280  fd 6e 79 73 2e d4 64 29  bf fc 53 70 ff d5 d5 17  |.nys..d)..Sp....|
00000290  18 1e aa c1 37 6f 90 d0  3b 8a da b7 00 9c af 1b  |....7o..;.......|
000002a0  93 ed 75 3b ee ff 38 1d  4b ca b7 64 b9 51 5e 7d  |..u;..8.K..d.Q^}|
000002b0  22 b5 a5 fa c1 57 21 5e  67 4b fa bb 24 48 83 8e  |"....W!^gK..$H..|
000002c0  68 ca b2 08 98 18 17 56  c4 61 e6 8b 1b 07 ea a8  |h......V.a......|
000002d0  71 30 1f 12 d9 67 b1 75  0f e9 84 8c ed af 44 2e  |q0...g.u......D.|
000002e0  47 0b ee 04 b7 0a b8 71  73 16 19 d0 9a 24 47 be  |G......qs....$G.|
000002f0  9e 1b 06 50 a6 34 52 e2  bf 10 2b 2c 95 9a d3 38  |...P.4R...+,...8|
00000300  7b 37 4c 50 a6 5d a1 da  f8 bd b3 3c 49 1b 0f 96  |{7LP.].....<I...|
00000310  7f c9 90 af 46 00 05 10  19 3b 1a 99 9a 01 62 8d  |....F....;....b.|
00000320  eb f4 d4 02 f5 67 0e e0  f2 1a 4e 4c 7c 89 fc fb  |.....g....NL|...|
00000330  ff 29 f0 1f 30 98 a4 48  83 cc a0 74 e6 3b 65 c0  |.)..0..H...t.;e.|

 

3.  The random data will typically have a repeating pattern, e.g. every 4 MB, as seen in the example below at offsets 000000c0, 004000c0, 008000c0.

  • For example we can see a repeating pattern :  ee 4f 26 6f 8d 37 b2 46
000000c0  4a 3d 54 c7 00 9e 7c ed  ee 4f 26 6f 8d 37 b2 46  |J=T...|..O&o.7.F|
000000d0  ef 6a 8e d0 66 f9 5a 18  15 e1 48 87 c2 a5 d9 24  |.j..f.Z...H....$|
000000e0  47 62 20 37 18 7d 47 95  0b bf bc 82 13 29 c2 93  |Gb 7.}G......)..|
000000f0  ec fa ad 62 08 fa bd 05  48 0f 26 6f 47 f9 1e d3  |...b....H.&oG...|
00000100  61 fe 52 86 52 c4 f2 b4  7e c1 ca fd 3d 03 f1 46  |a.R.R...~...=..F|
00000110  b4 ca 8d 3c 47 e7 5c f9  93 a5 ba d9 10 fa 69 61  |...<G.\.......ia|
00000120  94 ef 3c d2 7d 56 f9 ce  f8 ab 4d 81 2a 0a 8b 30  |..<.}V....M.*..0|
00000130  0c 7f 06 06 65 af d6 84  90 42 34 99 88 5a dc 11  |....e....B4..Z..|
00000140  c8 3f fc f7 b4 7d 0b c3  e3 c5 20 f0 1c 12 4e 03  |.?...}.... ...N.|
00000150  99 47 e9 8b 43 b2 38 90  73 f6 28 2f 80 b7 0a 00  |.G..C.8.s.(/....|
00000160  8b 51 3a 97 ca 3e e3 fc  04 c5 f0 d7 27 a1 35 9e  |.Q:..>......'.5.|
00000170  58 c5 20 91 d4 8e 96 2f  52 d8 fe b3 a6 65 62 2d  |X. ..../R....eb-|
00000180  43 c8 0d bc 0f 74 eb 3e  98 b1 18 0f 8d e4 81 36  |C....t.>.......6|
00000190  80 fa 96 d4 9b 0e ab cd  43 ff af 4f 05 c8 61 e0  |........C..O..a.|
000001a0  cd 77 63 90 0b 0c 7e 38  3f e0 f7 24 f6 3c b8 d8  |.wc...~8?..$.<..|
000001b0  5b de 55 17 b6 38 4b c4  ba 1f d2 c9 f8 ae c3 15  |[.U..8K.........|
000001c0  70 9a ad 7d 73 26 7e 3b  c7 e7 62 cb 01 88 96 49  |p..}s&~;..b....I|
...
...
004000c0  4a 3d 54 c7 00 9e 7c ed  ee 4f 26 6f 8d 37 b2 46  |J=T...|..O&o.7.F|
004000d0  ef 6a 8e d0 66 f9 5a 18  15 e1 48 87 c2 a5 d9 24  |.j..f.Z...H....$|
004000e0  47 62 20 37 18 7d 47 95  0b bf bc 82 13 29 c2 93  |Gb 7.}G......)..|
004000f0  ec fa ad 62 08 fa bd 05  48 0f 26 6f 47 f9 1e d3  |...b....H.&oG...|
00400100  61 fe 52 86 52 c4 f2 b4  7e c1 ca fd 3d 03 f1 46  |a.R.R...~...=..F|
00400110  b4 ca 8d 3c 47 e7 5c f9  93 a5 ba d9 10 fa 69 61  |...<G.\.......ia|
00400120  94 ef 3c d2 7d 56 f9 ce  f8 ab 4d 81 2a 0a 8b 30  |..<.}V....M.*..0|
00400130  0c 7f 06 06 65 af d6 84  90 42 34 99 88 5a dc 11  |....e....B4..Z..|
00400140  c8 3f fc f7 b4 7d 0b c3  e3 c5 20 f0 1c 12 4e 03  |.?...}.... ...N.|
00400150  99 47 e9 8b 43 b2 38 90  73 f6 28 2f 80 b7 0a 00  |.G..C.8.s.(/....|
00400160  8b 51 3a 97 ca 3e e3 fc  04 c5 f0 d7 27 a1 35 9e  |.Q:..>......'.5.|
00400170  58 c5 20 91 d4 8e 96 2f  52 d8 fe b3 a6 65 62 2d  |X. ..../R....eb-|
00400180  43 c8 0d bc 0f 74 eb 3e  98 b1 18 0f 8d e4 81 36  |C....t.>.......6|
00400190  80 fa 96 d4 9b 0e ab cd  43 ff af 4f 05 c8 61 e0  |........C..O..a.|
004001a0  cd 77 63 90 0b 0c 7e 38  3f e0 f7 24 f6 3c b8 d8  |.wc...~8?..$.<..|
004001b0  5b de 55 17 b6 38 4b c4  ba 1f d2 c9 f8 ae c3 15  |[.U..8K.........|
004001c0  70 9a ad 7d 73 26 7e 3b  c7 e7 62 cb 01 88 96 49  |p..}s&~;..b....I|
...
...
008000c0  4a 3d 54 c7 00 9e 7c ed  ee 4f 26 6f 8d 37 b2 46  |J=T...|..O&o.7.F|
008000d0  ef 6a 8e d0 66 f9 5a 18  15 e1 48 87 c2 a5 d9 24  |.j..f.Z...H....$|
008000e0  47 62 20 37 18 7d 47 95  0b bf bc 82 13 29 c2 93  |Gb 7.}G......)..|
008000f0  ec fa ad 62 08 fa bd 05  48 0f 26 6f 47 f9 1e d3  |...b....H.&oG...|
00800100  61 fe 52 86 52 c4 f2 b4  7e c1 ca fd 3d 03 f1 46  |a.R.R...~...=..F|
00800110  b4 ca 8d 3c 47 e7 5c f9  93 a5 ba d9 10 fa 69 61  |...<G.\.......ia|
00800120  94 ef 3c d2 7d 56 f9 ce  f8 ab 4d 81 2a 0a 8b 30  |..<.}V....M.*..0|
00800130  0c 7f 06 06 65 af d6 84  90 42 34 99 88 5a dc 11  |....e....B4..Z..|
00800140  c8 3f fc f7 b4 7d 0b c3  e3 c5 20 f0 1c 12 4e 03  |.?...}.... ...N.|
00800150  99 47 e9 8b 43 b2 38 90  73 f6 28 2f 80 b7 0a 00  |.G..C.8.s.(/....|
00800160  8b 51 3a 97 ca 3e e3 fc  04 c5 f0 d7 27 a1 35 9e  |.Q:..>......'.5.|
00800170  58 c5 20 91 d4 8e 96 2f  52 d8 fe b3 a6 65 62 2d  |X. ..../R....eb-|
00800180  43 c8 0d bc 0f 74 eb 3e  98 b1 18 0f 8d e4 81 36  |C....t.>.......6|
00800190  80 fa 96 d4 9b 0e ab cd  43 ff af 4f 05 c8 61 e0  |........C..O..a.|
008001a0  cd 77 63 90 0b 0c 7e 38  3f e0 f7 24 f6 3c b8 d8  |.wc...~8?..$.<..|
008001b0  5b de 55 17 b6 38 4b c4  ba 1f d2 c9 f8 ae c3 15  |[.U..8K.........|
008001c0  70 9a ad 7d 73 26 7e 3b  c7 e7 62 cb 01 88 96 49  |p..}s&~;..b....I|

Resolution

  • Identify any ongoing server decommissioning processes and stop them without delay. There may be further vmfs datastores whose LUNs are mapped to servers undergoing decommission which are at threat of being overwritten.
  • Unmap all shared datastores from servers before resuming decommissioning on them.
  • In most cases by the time this issue is detected the damage to the VMFS is extensive and have overwritten the datastore beyond the recover capability of VOMA, However to confirm this please contact support: Creating and managing Broadcom support cases

If the vmfs metadata is overwritten you will need to do one of the following:

  • Note:  If sdelete has completed on entire disk. Data recovery services will not be able to recover data.
Powered by Wolken Software