Unable to see the attached volume in instance

Asked by Shanthakumar K

Hi,

I have single node openstack setup where ( compute + controller ) installed and running with all the required services.

I am able to create the volume using cinder and assign to instance but unable to see the attached volume inside the Virtual instances

################################
Services
################################
keystone start/running, process 7740
glance-api start/running, process 7742
glance-registry start/running, process 7744
mysql start/running, process 7770
quantum-server start/running, process 7829
quantum-dhcp-agent start/running, process 7846
quantum-metadata-agent start/running, process 7851
quantum-l3-agent start/running, process 7866
quantum-plugin-linuxbridge-agent start/running, process 7871
 * Checking DNS forwarder and DHCP server dnsmasq * (dead, pid file exists)
nova-api start/running, process 8073
nova-cert start/running, process 8084
nova-compute start/running, process 8096
nova-conductor start/running, process 8107
nova-consoleauth start/running, process 8122
nova-novncproxy start/running, process 8134
nova-scheduler start/running, process 8152
libvirt-bin start/running, process 2035
Apache2 is running (pid 8247).
 * memcached is running

#####################################

 nova list
+--------------------------------------+--------------+--------+-------------------+
| ID | Name | Status | Networks |
+--------------------------------------+--------------+--------+-------------------+
| 3d34826f-480e-40bb-a2d5-9bd32b14ee05 | SB | ACTIVE | grzmrnw=50.50.1.4 |
| 0583f7a7-ed02-43b6-b3c7-c1d7405426b5 | Testvm-ciros | ACTIVE | grzmrnw=50.50.1.3 |
+--------------------------

cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 03d68746-6519-4f63-91d3-ba96603cf8cd | in-use | Testvol1 | 2 | None | false | 3d34826f-480e-40bb-a2d5-9bd32b14ee05 |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+

Nova Compute log:
#############

2013-04-12 07:17:40.957 8096 INFO nova.compute.manager [-] Updating host status
2013-04-12 07:17:42.408 AUDIT nova.compute.manager [req-2fd028df-8eda-48bd-a9ad-098074264bdb 4b106ff812b54472b8e2d3596524f730 fedfaa89548e41188c2dbd6f96d0de4a] [instance: 3d34826f-480e-40bb-a2d5-9bd32b14ee05] Attaching volume 03d68746-6519-4f63-91d3-ba96603cf8cd to /dev/vdb
2013-04-12 07:17:42.477 WARNING nova.virt.libvirt.utils [req-2fd028df-8eda-48bd-a9ad-098074264bdb 4b106ff812b54472b8e2d3596524f730 fedfaa89548e41188c2dbd6f96d0de4a] systool is not installed
2013-04-12 07:17:42.543 WARNING nova.virt.libvirt.utils [req-2fd028df-8eda-48bd-a9ad-098074264bdb 4b106ff812b54472b8e2d3596524f730 fedfaa89548e41188c2dbd6f96d0de4a] systool is not installed
2013-04-12 07:18:42.130 8096 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2013-04-12 07:18:42.387 8096 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 141987
2013-04-12 07:18:42.388 8096 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -11
2013-04-12 07:18:42.388 8096 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 22
2013-04-12 07:18:42.443 8096 INFO nova.compute.resource_tracker [-] Compute_service record updated for grzrc3:grzrc3

Virtual Machine:
############

Log into VM

iaasc@grzrc3:~$ ssh cirros@50.50.1.4
The authenticity of host '50.50.1.4 (50.50.1.4)' can't be established.
RSA key fingerprint is 76:65:41:41:04:a0:2e:29:b4:8a:f5:11:a5:87:64:10.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '50.50.1.4' (RSA) to the list of known hosts.
cirros@50.50.1.4's password:
$
$ fdisk -l
$

Kindly help me on this.

Question information

Language:
English Edit question
Status:
Answered
For:
OpenStack Compute (nova) Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
drolfe (drolfe) said :
#1

I have the same error message on a single node setup of grizzly

Revision history for this message
Shanthakumar K (shantha-kumar) said :
#2

whats the cloud image you are using ?... Have uou find the route cause for this issue ?

Revision history for this message
drolfe (drolfe) said :
#3

Sorry I've since rebuild my dev cloud. If I were you I would be dugging the iscsi side of things.

Is it possible that the databases think the volume is connected but the iscsi connection has been dropped. i.e. have you rebooted your all in one node with the vm's and volumes still attached ?

Revision history for this message
drolfe (drolfe) said :
#4

As per https://answers.launchpad.net/cinder/+question/222896

"""""
And this problem was solved after doing: "yum install sysfsutils"
it seems "systool" was missing..
"""""

but in ubuntu I'd try "sudo apt-get install sysfsutils" which should install systool for you

Revision history for this message
Shanthakumar K (shantha-kumar) said :
#5

Thanks for your response.

the Q/A https://answers.launchpad.net/cinder/+question/222896 its related to attachment issue.

I'm getting the log as attachment successful and the horizon & CLI also says as volume attached to instance.

cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 03d68746-6519-4f63-91d3-ba96603cf8cd | in-use | Testvol1 | 2 | None | false | 3d34826f-480e-40bb-a2d5-9bd32b14ee05 |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+

coming to systools installation. do I need to install in launched instance or in controller node..

bcoz my VM doesnt have external route and doesnt have internet connection.

Revision history for this message
drolfe (drolfe) said :
#6

I would try in install systools on the compute node, not the vm instance

Revision history for this message
shu, xinxin (xinxin-shu) said :
#7

i meet the same problem, but i have installed systool on all compute nodes, no error message can be found in compute and volume log.

Revision history for this message
Shanthakumar K (shantha-kumar) said :
#8

Shu, are you able to see the volume after your systool installation

Revision history for this message
Hrushikesh (hrushikesh-gangur) said :
#9

In my environment, after installing sysfsutil tools on compute node, i saw warning message went away from nova-compute.log. However, fdisk -l on the launched instance still does not show the recently attached volume. Any ideas?

Revision history for this message
shu, xinxin (xinxin-shu) said :
#10

In my setup, my glance image does not load ‘acpiphp’ module to support hot plug, so after i loaded this module , everything is ok

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Hrushikesh
Sent: Friday, May 17, 2013 2:17 PM
To: Shu, Xinxin
Subject: Re: [Question #226471]: Unable to see the attached volume in instance

Question #226471 on OpenStack Compute (nova) changed:
https://answers.launchpad.net/nova/+question/226471

Hrushikesh posted a new comment:
In my environment, after installing sysfsutil tools on compute node, i saw warning message went away from nova-compute.log. However, fdisk -l on the launched instance still does not show the recently attached volume. Any ideas?

--
You received this question notification because you are a direct subscriber of the question.

Revision history for this message
drolfe (drolfe) said :
#11

I wrote up a blog post really for my own reference but you might as well have a look if your having issues with cinder as it covers the basics

http://drolfe.github.io/drolfe/blog/2013/05/01/troubleshoot-cinder-volumes/

Revision history for this message
Hrushikesh (hrushikesh-gangur) said :
#12

Thanks Xinxin and Daniel. Those information were really useful. In addition, here is what my observations were:

The confusion was that if you attach a device /dev/vdc, it used to get detected as /dev/vdb. So, it looks like when you attach a device with any path (/dev/vdc or /dev/vdd), OpenStack takes care of attaching with next device in the loop. For example, if you select a flavor m1.small, the devices that gets created by default are /dev/vda (root) and /dev/vdb (ephemeral). So, when you attach disk to that instance, it will show up as /dev/vdc (regardless of which device name is suggested while attaching). Seems to be a bug, but we know now how it works.

Revision history for this message
Zhuohua Li (zhuohual) said :
#13

use "sudo fdisk -l" you can see the attached disk

Revision history for this message
Mohan L (thefossgeek) said :
#14

I am using the VMDK image pointed below:

http://partnerweb.vmware.com/programs/vmdkimage/debian-2.6.32-i686.vmdk

https://wiki.openstack.org/wiki/NovaVMware/DeveloperGuide

Uploaded to Glnace:

# glance image-create --name Debian-Modified-Img --is-public=True --container-format=bare --disk-format=vmdk --property vmware-disktype="preallocated" < debian-2.6.32-i686.vmdk

1). Launch Instance works fine.

2). Getting internal IP to eth0 with adapter_type "lsiLogic" and and disk_type "preallocated".

3). Floating ip association works fine.

4). Able to ssh to VM.

5). Able to take VM console on Dashboard.

6). Able to create volume.

7). Able to attach volume when instance is running.

8). Unable to see the attached volume in instance.

# fdisk -l

Disk /dev/sda: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ae3eb

   Device Boot Start End Blocks Id System
/dev/sda1 * 1 119 949248 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 119 131 96257 5 Extended
Partition 2 does not end on cylinder boundary.
/dev/sda5 119 131 96256 82 Linux swap / Solaris

# cinder list
+--------------------------------------+--------+--------------+------+---------------+----------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+--------------+------+---------------+----------+--------------------------------------+
| a4c7ca6f-402c-4f54-bde5-0e6b6329fed9 | in-use | Vol2 | 1 | thick_volume2 | false | 2546eda7-8f92-47bd-8575-f6c26d895484 |

# nova list
+--------------------------------------+-----------------+--------+------------+-------------+--------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-----------------+--------+------------+-------------+--------------------------------------+
| 2546eda7-8f92-47bd-8575-f6c26d895484 | Debian-Modified | ACTIVE | - | Running | sravan-net1=2.2.3.36, <external_ip> |

I sysfsutils is already installed in Compute Node:

# yum install sysfsutils
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Setting up Install Process
Package sysfsutils-2.1.0-7.el6.x86_64 already installed and latest version

I have the following kernel modules loaded at boot time:

# lsmod
Module Size Used by
acpiphp 13281 0
loop 9777 0
snd_pcm 47226 0
snd_timer 12270 1 snd_pcm
psmouse 44845 0
snd 34431 2 snd_pcm,snd_timer
soundcore 3450 1 snd
evdev 5609 2
snd_page_alloc 5061 1 snd_pcm
serio_raw 2916 0
parport_pc 15799 0
shpchp 21200 0
parport 22554 1 parport_pc
container 1833 0
i2c_piix4 7076 0
pcspkr 1207 0
processor 26343 0
button 3598 0
ac 1636 0
pci_hotplug 18545 2 acpiphp,shpchp
i2c_core 12795 1 i2c_piix4
ext3 94536 1
jbd 32449 1 ext3
mbcache 3762 1 ext3
sd_mod 26029 3
crc_t10dif 1012 1 sd_mod
mptspi 9617 2
mptscsih 14196 1 mptspi
mptbase 43012 2 mptspi,mptscsih
ata_generic 2247 0
scsi_transport_spi 14874 1 mptspi
ata_piix 17744 0
libata 115925 2 ata_generic,ata_piix
floppy 40923 0
thermal 9206 0
thermal_sys 9378 2 processor,thermal
e1000 77357 0
scsi_mod 105157 5 sd_mod,mptspi,mptscsih,scsi_transport_spi,libata

However, fdisk -l on the launched instance still does not show the recently attached volume.

After instance reboot I am able to see attached volume:

# fdisk -l

Disk /dev/sda: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000ae3eb

   Device Boot Start End Blocks Id System
/dev/sda1 * 1 119 949248 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 119 131 96257 5 Extended
Partition 2 does not end on cylinder boundary.
/dev/sda5 119 131 96256 82 Linux swap / Solaris

Disk /dev/sdb: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table

Any ideas?

Can you help with this problem?

Provide an answer of your own, or ask Shanthakumar K for more information if necessary.

To post a message you must log in.