Fixing LVM I/O Errors

Situation :

Most of us faced errors mentioned below during our system admin activities   , this is mainly related to the removable storage media that we use on unix servers’

Reason of this could be because removing the disk/LUN’s without clean shutdown/unmount or de-attaching disks from LV’s  .

/dev/sdf: read failed after 0 of 4096 at 0: Input/output error

/dev/sdf: read failed after 0 of 4096 at 3298534817792: Input/output error

/dev/sdf: read failed after 0 of 4096 at 3298534875136: Input/output error

/dev/sdf: read failed after 0 of 4096 at 4096: Input/output error

/dev/sdk: read failed after 0 of 4096 at 0: Input/output error

/dev/sdk: read failed after 0 of 4096 at 6442385408: Input/output error

/dev/sdk: read failed after 0 of 4096 at 6442442752: Input/output error

/dev/sdk: read failed after 0 of 4096 at 4096: Input/output error

 Solution :

1) Check which Volume Group have the issue , run “vgscan” command .

2) Find out the Logical Volumes attached with that Volume Group .

3) Inactive the logical volumes as :

  # lvchange -an <lv-name>

4) Inactive Volume group as :

 #  vgchange -an <vg-name>

5) Again Scan Volume group using “vgscan” .

6) Now activate the Volume Group :

# vgchange -ay <volume-group-name>

7) Run command “lvscan” , the error should be gone now .

8) Now activate the Logical Volume Name :

# lvchange -ay <lv-name>

Note : In some cases if we want to use same media device again which was removed unclean but its still giving error after attaching then  we need to deattach the removable device manually  for a while  and then attach again , later  follow above steps .

12 thoughts on “Fixing LVM I/O Errors”

  1. Thank you so much! This worked on a SLES system and it was fixed faster after waiting over 6 hours for the vendor to call back!

    Reply
  2. well the procedure won’t help much on live system where you don’t have downtime. anyway, i tried lvchange -an with input/output error. following lvremove with input/output error, and the last step vgreduce –removemissing ,
    NOTE: on live system…
    Hope this help

    Reply
  3. A simplier and secure alternative to kpartx -d:

    # lvscan
    /dev/sde: read failed after 0 of 4096 at 0: Input/output error
    ACTIVE ‘/dev/vg1/lv1’ [450.00 GB] inherit

    # echo 1 > /sys/block/sde/device/delete
    # lvscan
    ACTIVE ‘/dev/vg1/lv1’ [450.00 GB] inherit

    Reply
  4. Hi,

    I forgot to unmount my multipath device and i have removed multipath device. Now i am getting I/O error. How to remove Logical volume, Volume Group, Physical volume now. OS version is Redhat 6.10

    REgards,
    Suresh Babu T

    Reply

Leave a Reply to Syed Kazmi Cancel reply