Posts Tagged lun
Adding a LUN for ASM Use to a RHEL 4/PowerPath Database Server
Posted by Jay Caviness in ASM on September 12th, 2008
This procedure assumes using the Qlogic drivers and PowerPath 4.5.1, all installed. In a RAC, the steps are divided into “first node” steps , “Remaining Node(s)” steps and “Updating the ASM Instance”. For a standalone on SAN database server, the “Remaining Node(s)” step can be skipped.
Whenever performing a procedure like this one that will alter SAN/Disk layout on the fly, make sure that you have adequate backups of the database and the system in the unlikely event that something goes wrong.
First Node Steps:
Choose one node where all of the initial work will be done. Later a subset of these steps will be applied to the other cluster nodes.
1. Using the SAN software, present the new LUNS to the Linux nodes.
2. Get a list of the current visible LUNs for later reference using powermt, or a wrapper like:
# /stage/mck_ks/bin/map_pp_pseudo2lun /dev/emcpowera = LUN 165 /dev/emcpowerb = LUN 330 /dev/emcpowerc = LUN 423 /dev/emcpowerd = LUN 422 /dev/emcpowere = LUN 420 /dev/emcpowerf = LUN 331 /dev/emcpowerg = LUN 421 /dev/emcpowerh = LUN 335
3. Reboot the node (or rescan the SCSI bus)
a. Rebooting is the supported method to rescan the SCSI bus
b. To rescan (experimental), run: /root/rescan_for_luns
4. Using powermt, (or its wrapper), identify the new LUNs:
# /stage/mck_ks/bin/map_pp_pseudo2lun /dev/emcpowera = LUN 165 /dev/emcpowerb = LUN 330 /dev/emcpowerc = LUN 423 /dev/emcpowerd = LUN 422 /dev/emcpowere = LUN 420 /dev/emcpowerf = LUN 331 /dev/emcpowerg = LUN 421 /dev/emcpowerh = LUN 335 /dev/emcpoweri = LUN 369
5. Partition the new LUNs so they can be used for ASM. For EACH new LUN added run the following (the newlines matter!):
cat < < EOF | fdisk /dev/emcpower<newdevice> n p 1 1 w EOF
6. Using powermt, identify the “path” devices. These are the “sdXX” devices in the output below. In the next step we’ll change the ownership of these devices.
# powermt display dev=emcpower<device> Pseudo name=emcpoweri CLARiiON ID=APM00031200473 [SG: HA Linux RAC 10g] Logical device ID=6006016069C30A0004A2400E993ADA11 [LUN 369] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP B ========================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ========================================================== 1 qla2xxx sdah SP B2 active alive 0 0 1 qla2xxx sdai SP A1 active alive 0 0 2 qla2xxx sdaj SP A3 active alive 0 0 2 qla2xxx sdak SP B1 active alive 0 0</device>
7. Now using the pseudo device name and the “path” device names change the ownership so Oracle can use them and change the Udev permissions file, so these are owned correctly after reboots (the device names below are samples taken from the output above):
for i in emcpoweri sdah sdai sdaj sdak
do
chown oracle:dba /dev/${i}1
echo "${i}1:oracle:dba:0660" >> /etc/udev/permissions.d/10-oracle-devices.permissions
done8. Using the oracleasm command, determine the labels for the current ASM disks. The new disks will generally labeled in the same pattern with the next highest numbers:
# service oracleasm listdisks VOL001 VOL002 VOL003 VOL004
9. Now label the new disk(s) with the oracleasm command. RE-run and increment the label until all the new disks are done (the example below added emcpoweri1).
# service oracleasm createdisk VOL005 /dev/emcpoweri1 Marking disk "/dev/emcpoweri1" as an ASM disk: [ OK ]
10. Now confirm that the new disks are labeled using the ‘oracleam listdisks’ command. The new disk should show up.
# service oracleasm listdisks VOL001 VOL002 VOL003 VOL004 VOL005
Remaining Node(s) Steps:
These steps need to be performed on ALL the remaining RAC nodes AFTER the above has been completed. For standalone DB servers, skip these steps.
1. Using the SAN software, present the new LUNS to the Linux nodes.
2. Get a list of the current visible LUNs for later reference using powermt, or a wrapper like:
# /stage/mck_ks/bin/map_pp_pseudo2lun /dev/emcpowera = LUN 165 /dev/emcpowerb = LUN 330 /dev/emcpowerc = LUN 423 /dev/emcpowerd = LUN 422 /dev/emcpowere = LUN 420 /dev/emcpowerf = LUN 331 /dev/emcpowerg = LUN 421 /dev/emcpowerh = LUN 335
3. Reboot the node (or rescan the SCSI bus)
a. Rebooting is the supported method to rescan the SCSI bus
b. To rescan (experimental), run: /root/rescan_for_luns
4. Using powermt, (or its wrapper), identify the new LUNs:
# /stage/mck_ks/bin/map_pp_pseudo2lun /dev/emcpowera = LUN 165 /dev/emcpowerb = LUN 330 /dev/emcpowerc = LUN 423 /dev/emcpowerd = LUN 422 /dev/emcpowere = LUN 420 /dev/emcpowerf = LUN 331 /dev/emcpowerg = LUN 421 /dev/emcpowerh = LUN 335 /dev/emcpoweri = LUN 369
5. Using powermt, identify the “path” devices. These are the “sdXX” devices in the output below. In the next step we’ll change the ownership of these devices.
# powermt display dev=emcpower<device> Pseudo name=emcpoweri CLARiiON ID=APM00031200473 [SG: HA Linux RAC 10g] Logical device ID=6006016069C30A0004A2400E993ADA11 [LUN 369] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP B ========================================================== ---------------- Host --------------- - Stor - -- I/O Path - -- Stats --- ### HW Path I/O Paths Interf. Mode State Q-IOs Errors ========================================================== 1 qla2xxx sdah SP B2 active alive 0 0 1 qla2xxx sdai SP A1 active alive 0 0 2 qla2xxx sdaj SP A3 active alive 0 0 2 qla2xxx sdak SP B1 active alive 0 0</device>
6. Now using the Pseudo device name and the “path” device names change the ownership so Oracle can use them and change the Udev permissions file, so these are owned correctly after reboots (the device names below are samples taken from the output above):
for i in emcpoweri sdah sdai sdaj sdak
do
partprobe /dev/$i
chown oracle:dba /dev/${i}1
echo "${i}1:oracle:dba:0660" >> /etc/udev/permissions.d/10-oracle-devices.permissions
done7. Using the oracleasm command, determine the labels for the current ASM disks. The new disks will generally labeled in the same pattern with the next highest numbers:
# service oracleasm listdisks VOL001 VOL002 VOL003 VOL004
8. Now scan the ASM disks to add the new ones:
# service oracleasm scandisks Scanning system for ASM disks: [ OK ]
9. Now confirm that the new disks are labeled using the ‘oracleam listdisks’ command. The new disk should show up.
# service oracleasm listdisks VOL001 VOL002 VOL003 VOL004 VOL005
Updating the ASM instance:
All of these steps must be performed as the oracle user and only AFTER the above steps have been successfully completed for ALL nodes in the cluster. In a RAC cluster, this step needs to be completed on only ONE node in the cluster; since the database is shared, the others will get the change.
1. Make sure you set your SID to ‘+ASM’:
/home/oracle:(+ASM)$ . oraenv ORACLE_SID = [+ASM] ? +ASM
2. Log in to the ASM instance as sysdba.
/home/oracle:(+ASM)$ sqlplus "/ as sysdba"
3. Now use the following command to add the new ASM disks to the disk group. Notice you can use some vary basic regular expressions in the naming of the volumes. In the case below, we added ASM VOL002, VOL003, VOL004. Don’t try to add already existing volumes, otherwise the whole command will error out, so customize the volume name portion below to reflect your naming scheme for the new LUNs.
SQL> alter diskgroup dgroup1 add disk 'ORCL:VOL00[234]'; Diskgroup altered.
4. You can verify the new disk by running the following SQL script:
SQL> @/home/oracle/asm/asm_disks.sql
DiskGroup NamePath FileName FailGroup FileSize(MB) UsedSize(MB) Pct.Used
--------------- ----------------- -------------------- ------------------
DGROUP1 ORCL:VOL001 VOL001 VOL001 15,359 7,338 47.78
ORCL:VOL002 VOL002 VOL002 51,199 2,525 4.93
ORCL:VOL003 VOL003 VOL003 51,199 2,525 4.93
ORCL:VOL004 VOL004 VOL004 51,199 2,525 4.93
-------------- --------------
Grand Total: 168,956 14,913