[3.0]The disk drive for x is not yet ready or not present & hang after fsck

[Commifreak]

UPDATE: Its a kernel-related problem! 3.2.0-31 is causing this problem! 3.2.0-29 is working well!

See also: http://ubuntuforums.org/showthread.php?p=12264791

==============================[/b]

Hi

I've successfully set up Zentyal with the latest installer (x32).

The complete installation procedure finished successfully.

Here my hard drive specs:

4x 2TB drives (SATA, AHCI)

3 RAIDS (1x swap, 1x /boot 1x raid5 for LVM) (not created, it was already present from the previous Zentyal version)
2 VG's: Mountpoint / and /home

Until now, I've worked with previous Zentyal stable version.

All I did to switch to Zentyal 3: leave the /home as is and format all other partitions. The Zentyal installer has recognized all raid and LVM devices. This config seems to run successfully.
Since the fresh install (no, wait - 3 until now ) I have one mysterious issue:

Every boot, I get this (https://www.dropbox.com/s/5vy5zkgbvs7p5co/20120914_213014.jpg) error. But only for a few seconds.
I cant imagine what is going wrong...

I've installed the 5 system updates which are available directly after installation and rebooted the system. Now I get a hang on every boot:
fsck messages are printed along with the messages which LVMs are present. After these few lines, nothing happened. 2 minutes later I noticed the error "Could not write bytes: broken pipe", but the system starts in both cases without any following error.

The syslog doesn say anything, all normal.

These error was reproduceable every boot.

Fresh install again (with same partitions): Exactly same behaviour: After install: The error message that one device is not ready. And only after install the 5 system updates (linux-firmware, new kernel and some other things) I get the hang.

Maybe Ubuntu doesnt like the existing RAID/LVM config?

Should I clean all disks (low level format) and re-create the config, using the Installer?

Maybe someone has a hint for me

EDIT: The concerned drive is the RAID for the swap partition. This swap partition cames from the previous installation (not formatted during Zentyal 3 installation). Can this produce the "hang"?

Edit2: Maybe Useful resources on this issue?:

• https://bugs.launchpad.net/ubuntu/+source/mountall/+bug/776908
• http://ubuntuforums.org/showthread.php?t=1772692

It seems to be a related problem to the issue on the posted pages.

I'll try a few things to get it working again.

[Commifreak]

Ok, I figured the one thing out:

fstab:

Code: [Select]

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
/dev/mapper/VG1-System /               ext4    errors=remount-ro 0       1
# /boot was on /dev/md127 during installation
UUID=6bd29ed7-06ec-422c-a39e-dac8a3b97d9e /boot           ext2    defaults        0       2
/dev/mapper/VG1-Data /home           ext4    defaults        0       2
/dev/md125      none            swap    sw              0       0

mdadm.conf:

Code: [Select]

# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md/0 metadata=1.2 UUID=fa658e2d:9cf50829:dc877c7d:68e5af77 name=server:0
ARRAY /dev/md/1 metadata=1.2 UUID=127f876f:d06c6640:10e117cb:9eefbbf5 name=server:1
ARRAY /dev/md/2 metadata=1.2 UUID=2e48847e:e825e289:b0463acd:d643e598 name=server:2

# This file was auto-generated on Fri, 14 Sep 2012 21:00:00 +0200
# by mkconf $Id$

As you can see, fstab is trying to mount /dev/md125 which is not provided.

I set this line to /dev/md1, which is the swap partition.
Another thing: Why is fstab configured with this non-existing partition? These drives are detected by the partition manager during installation (manual mode was showing this md-drives with these 3 numbers after 'md').

The error, that the drive is not yet ready has gone, but the hang still persist.

On every boot, Zentyal hangs at:

Code: [Select]

fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
fsck from util-linux 2.20.1
/dev/mapper/VG1-System: clean, 88435/3055616 files, 558454/12206080 blocks
/dev/md0: clean, 230/488640 files, 108855/975860 blocks
/dev/mapper/VG1-Data: clean, 27599/181342208 files, 645157851/1450730496 blocks

It simply freezes. After 2-3 minutes, the boot continues without any error. All is functional.

Can someone explain this?

[jsalamero]

does it work when you use plain ubuntu server installer?

[Commifreak]

does it work when you use plain ubuntu server installer?

No, tried that today with latest Ubuntu Server installer. Same issue but with one more error before login: mountall: Plymouth command failed
mountall: disconnected from Plymouth

I dont know whats the reason. Maybe the stored raid-metadata on the disks from 2010 (last Zentyal Ubuntu-ground) is too old?

I've backed up all data and will format all disks. After that, I'll set up a fresh partition-table & RAID/LVM config.

[Commifreak]

Okay - Now I'm confused:

• cleared all drives using DBAN - MBR was empty, too
• Started installation from latest Zentyal stable iso image - burned onto a CD and installed successfully with folliowing config:

4 x 2TB Hard drives:
    * 3 raids per drive (2x raid1, 1x raid5)
    * 1st raid1 used as swap (2GB size)
    * 2nd raid1 used as /boot (1GB, ext2)
    * raid5 used as LVM member:

LVM:
    * created 1 VG: VG1
    * created 50GB LV: LV-System used as mountpoint /
    * created remaining space as LV: LV-Data used as mountpoint /home

Thats it - installation was successfully.

EVERY boot it hangs on the same step: after the 6 fsck lines for about 2 minutes. No error, nothing. Even no "The disk drive for x is not..." - simply nothing!

I really dont know where to begin to search, so I post my complete syslog below.

I hope someone finds the source of all this trouble.

EDIT:
    * syslog: https://www.dropbox.com/s/k5tb96id5ij1nxo/syslog.txt
    * CD-check was successfull (Ubuntu found no errors, valid disk)
    * fstab: https://www.dropbox.com/s/jwejjvm5gswvm23/fstab.txt
    * mdadm.conf: https://www.dropbox.com/s/ft2ydw49ciu6m95/mdadm.conf.txt

EDIT2:

Maybe plymouth wont play with me?

--> Sep 26 15:52:40 server kernel: [  412.022565] init: plymouth-stop pre-start process (6918) terminated with status 1

[Commifreak]

Finally, I gave up.

Maybe its really a bug, maybe on fsck: https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/945075

I'll reinstall the system (again) and will live with this bug, for the moment - it hurts nothing else, all is working, except fsck.

Maybe someone else has a tip for me

BTW: The initially error was related to the wrong detection of my raid: After my clean install today, the fstab was correct, no error (thread title) was shown.

[Commifreak]

I've forgot an important thing: I've noticed, that the hang was after the kernel-upgrade to 3.2.0-31 (Zentyalinstaller provides -29) but I've forgot that :/

So, it seems to be maybe a kernel-related issue.

--> http://ubuntuforums.org/showpost.php?p=12264791&postcount=4

[jjm1982]

I believe part of the problem is that grub doesn't like being installed on any type of RAID. I had something similar to this a while ago when I set my servers up. I ended up installing a flash drive and installed grub on that (/boot partition).

I'm still running Zentyal 2.2; waiting patiently for the steps to upgrade to 3.0.

[Commifreak]

I believe part of the problem is that grub doesn't like being installed on any type of RAID. I had something similar to this a while ago when I set my servers up. I ended up installing a flash drive and installed grub on that (/boot partition).

I'm still running Zentyal 2.2; waiting patiently for the steps to upgrade to 3.0.

I've thought the same, but its safe on a RAID1 (raid1 is supported in grub, raid5 would be more problematic).

Zentyal 2.2 was running fine on a completely raid1, with grub on it. I dont believe that GRUB is causing this.

[jjm1982]

That may have been my problem. I believe I wanted to have an all RAID 5 solution but ran into issues with grub on the disks. I didn't attempt the RAID one because I wanted to distribute all available disk space evenly.


fstab
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/md2 during installation
UUID=fdb0a3cf-7233-4275-ae65-2a9ced10e8e5 /               ext3    errors=remount -ro 0       1
# /boot was on /dev/sde1 during installation
UUID=78525e51-e3ba-4dc3-a47a-4cf17e38e120 /boot           ext3    defaults   0       2
# /home was on /dev/md3 during installation
UUID=f2a3f316-b5e9-43f1-92ce-03f971135fc8 /home           ext3    defaults,usrquota,grpquota,acl        0       2
# swap was on /dev/md1 during installation
UUID=ff745780-be17-4c4d-97f1-e294e30274d2 none            swap    sw  0       0

# /mnt/md0 was added to support larger shares
UUID=137c11d2-c39d-4b63-a439-b18ce6663126 /mnt/md0        ext3    defaults  0  0

# /mnt/md5 was added to support some type of backup
#UUID=1c268a08:1aec47ae:12179e22:45b82d75 /mnt/md5        ext3    defaults  0  0

#/dev/md5       /mnt/md5          ext3    defaults        0       0

mdadm.conf
# mdadm.conf
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default, scan all partitions (/proc/partitions) for MD superblocks.
# alternatively, specify devices to scan, using wildcards if desired.
DEVICE partitions

# auto-create devices with Debian standard permissions
CREATE owner=root group=disk mode=0660 auto=yes

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays
ARRAY /dev/md2 level=raid5 num-devices=4 UUID=43aa8d6c:2657c609:714bd8f1:e44e3d8
2
ARRAY /dev/md1 level=raid0 num-devices=4 UUID=21082e1a:57899df7:bcd3596e:7ab75f9
1
ARRAY /dev/md3 level=raid5 num-devices=4 UUID=84948efa:19a28bcf:1c79f79a:436bb9a
3
ARRAY /dev/md0 level=raid5 num-devices=4 UUID=a13386fb:153f2d6a:5efd6a88:1223ece
1
# This file was auto-generated on Thu, 06 Oct 2011 18:40:37 -0400
# by mkconf $Id$

df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2               23G  7.1G   14G  34% /
none                  940M  356K  939M   1% /dev
none                  944M     0  944M   0% /dev/shm
none                  944M  848K  943M   1% /var/run
none                  944M     0  944M   0% /var/lock
none                  944M     0  944M   0% /lib/init/rw
none                   23G  7.1G   14G  34% /var/lib/ureadahead/debugfs
/dev/md0              6.8T  1.8T  4.7T  27% /mnt/md0
/dev/sdb1             461M  157M  281M  36% /boot
/dev/md3              5.4T  3.0T  2.2T  58% /home

[Commifreak]

Hi, again.

I've set up the system again. I can confirm, that the kernel (3.2.0-31) is causing the problem! I dont know whats changed between -30 and -31 but something is buggy there

I'll use the -29 until a fixed version is released.