/dev/sda1 has gone 255 days without being checked, check forced

/dev/sda1 has gone 255 days without being checked, check forcedOn initial boot of a freshly deployed (Ubuntu) Linux appliance on VMware vSphere 5 the machine stopped working because of an inconsistent disk. According to the machine 255 days where passed without the disk being checked.  After a file system check is forced the machine stops with an error UNEXPECTED INCONSISTENCY followed by fsck / [5001] terminated with status 4.

/dev/sda1 has gone x days without being checked

The first message was very strange since the machine was freshly deployed from an OVF. According to the vendor this issue occurs when the date and time on the Hypervisor Host is wrong. After correcting the date and time of the hypervisor to the current date and time this problem should be resolved.

I verified the date and time of the ESX host, virtual machine and my laptop; they all matched (although the ESX host and virtual machine show the UTC time and I run in DST (+1). Since this problem occurred on October 29th 2013, I tried changing the time to the past hoping to trick the machine. During the attempts I found the Superblock last mount time of the devices:

  • /dev/sda1 – Fri Feb 15 10:53:23 2013
  • /dev/sda6 – Fri Aug 16 18:32:41 2013

The results of my test where unsuccessful, but it’s worth sharing anyway

Attempt #1  – January 1st 2013

  • /dev/sda1 – Superblock last mount time (Fri Feb 15 10:53:34 2013, now = Tue Jan 1 14:49:08 2013) is in the future
  • /dev/sda6 – Superblock last mount time (Fri Aug 16 19:32:31 2013, now = Tue Jan 1 14:49:08 2013) is in the future

Attempt #2 – May 15th  2013

  • /dev/sda1 – Clean
  • /dev/sda6 – Superblock last mount time (Fri Aug 16 19:32:31 2013, now = Tue Jan 1 14:49:08 2013) is in the future

Attempt #3 – August 8th  2013

  • /dev/sda1 – Clean
  • /dev/sda6 – Superblock last mount time (Fri Aug 16 19:32:31 2013, now = Thu Aug 18 15:07:31 2013) is in the future

Attempt #4 – August 15th 2013

  • /dev/sda1 – /dev/sda1 gas gone 180 days without being checked, check forced
  • /dev/sda6 – Superblock last mount time (Fri Aug 16 19:32:31 2013, now = Thu Aug 15 18:33:05 2013) is in the future

Attempt #5 – August 16th 2013

  • /dev/sda1 – /dev/sda1 gas gone 182 days without being checked, check forced
  • /dev/sda6 – /dev/sda6 gas gone 182 days without being checked, check forced

Attempt #6 – December 12th 2013

  • /dev/sda1 – /dev/sda1 gas gone 299 days without being checked, check forced
  • /dev/sda6 – /dev/sda6 gas gone 299 days without being checked, check forced

Attemp - August 15th 2013

As you can see I tried a lot (while not all of them made sense, I agree) but I could not find the sweet spot. Time for the next approach!

 

fsck – file system check

The machine clearly indicated two problems:

  1. The last file system check was executed 255 days ago
  2. There are errors with the consistency of the disk (filesystem has errors)

Since I found no other solution I addressed both problems at the same time: run a file system check.

Since the machine halted I was unable to run any command, including fsck,. So I mounted a Linux Live CD (FedoraLiveCD in my case, but you could also use the Ubuntu LiveCD) and ensured it booted from CD (change the boot order). After starting a shell (Terminal) I ran the su command to elevate my permissions.

su

Next I ran the fsck command on the /dev/sda1 device with the –f argument  to force checking of the file system and the –y argument to answer Yes on all questions.

fsck /dev/sda1 -f –y

fsck /dev/sda1 -f -y

And the same command but for the the /dev/sda6device,.

fsck /dev/sda6 -f -y

fsck /dev/sda6 -f -y

Success

Sure enough (after unmounting the ISO) the machine now boots succesfull.

 

 

.

Was once an enthusiastic PepperByte employee but is now working at Citrix. His blogs are still valuable to us and we hope to you too.