
COMSTAR setup with Sun Fire x4270 M2 with 12 SAS drives, Sun Storage
6 Gb SAS REM HBA, and Sun StorageTek 8 Gb FC PCIe HBA.

################################################################################
Configure two 5-way stripes on the 2TB SAS disks using either the HW RAID
interface from the Sun Fire X4270 M2 BIOS menu or the Pre-boot CLI.
This assumes that the Operating System is installed on target 0.

  Commands used with the Pre-boot CLI to create the Logical Drives.
  Create Logical Drives with a 5-wide stripe. The enclosure number is
  reported as the "Device ID" for Enclosure 0 using the following:

  % megacli -EncInfo -aALL

  The following example creates a 5-wide stripe on disks 2,3,6,8,9 with
  options: Writeback cache enabled, no readahead, no cache enabled with bad
  batteries, and a stripe size of 256K on the device with Enclosure ID
  number = 28:
  % megacli -CfgLDAdd -R0 [28:2,28:3,28:6,28:8,28:9] WB NORA Cached NoCachedBadBBU -strpsz 256 -a0

  Repeat for second set of 5 disks:
  % megacli -CfgLDAdd -R0 [28:4,28:5,28:7,28:10,28:11] WB NORA Cached NoCachedBadBBU -strpsz 256 -a0

  Format output should be similar to the following:

AVAILABLE DISK SELECTIONS:
       0. c7t0d0 <LSI cyl 60754 alt 2 hd 255 sec 252>
          /pci@0,0/pci8086,340c@5/pci1000,9263@0/sd@0,0
       1. c7t1d0 <LSI cyl 60755 alt 2 hd 255 sec 252>
          /pci@0,0/pci8086,340c@5/pci1000,9263@0/sd@1,0
       2. c7t2d0 <LSI-MR9261-8i-2.90-9.09TB>
          /pci@0,0/pci8086,340c@5/pci1000,9263@0/sd@2,0
       3. c7t3d0 <LSI-MR9261-8i-2.90-9.09TB>
          /pci@0,0/pci8086,340c@5/pci1000,9263@0/sd@3,0

Documentation for use in configuring internal RAID devices available at:
http://www.oracle.com/technetwork/documentation/oracle-x86-servers-190077.html
1. Sun Fire X4170 M2 and X4270 M2 Servers Installation Guide,
   Part No.: 821-0481-15
2. Sun Fire X4270 M2 Server Service Manual, Part No. E21671-02

################################################################################
Configure the QLC driver to run in target mode to prepare the x4270 M2 to be
a COMSTAR server.
################################################################################
  Enable stmf services
  % svcadm enable stmf

  Verify QLC device contains pciex1077,2532
  % echo ::devbindings -q qlc | mdb -k
  ffffff02d280f580 pciex1077,2532, instance #0 (driver name: qlc)
  ffffff02d280f2d0 pciex1077,2532, instance #1 (driver name: qlc)

  Notify the system to replace the qlc driver with the the qlt (Qlogic
  Target driver) for this device.
  % cp /etc/driver_aliases /etc/driver_aliases.orig
  % update_drv -d -i '"pciex1077,2532"' qlc
  % nawk '!($1 == "qlc" && $2 == "\"pciex1077,2532\"")' /etc/driver_aliases > /etc/driver_aliases.new
  % cp /etc/driver_aliases.new /etc/driver_aliases
  % update_drv -a -i '"pciex1077,2532"' qlt
  devfsadm: driver failed to attach: qlt
  Warning: Driver (qlt) successfully added to system but failed to attach

  Reboot the system to complete qlt driver install
  % reboot

  After reboot check that qlc devices are now in target mode
  where the qlt driver is loaded:

  %  echo "::devbindings -q qlt" | mdb -k
  ffffff024eb84580 pciex1077,2532, instance #0 (driver name: qlt)
  ffffff024eb842d0 pciex1077,2532, instance #1 (driver name: qlt)

  % fcinfo hba-port | egrep 'HBA|Mode|Driver Name'
  HBA Port WWN: 21000024ff303b22
        Port Mode: Target
        Model: QLE2562
        Driver Name: COMSTAR QLT
  HBA Port WWN: 21000024ff303b23
        Port Mode: Target
        Model: QLE2562
        Driver Name: COMSTAR QLT

Note: the WWN numbers listed by fcinfo will be different
than those listed above.

################################################################################
Apply labels and partition the HW RAID devices:
################################################################################

  Install an EFI label on the 2 9TB Logical Drives:
  % format -e c7t2d0 
  > fdisk
    select 3: delete any existing partitions
    select 1: create Solaris2 partition type using 100% of disk
    select 6: exit with updating disk configuration
  > label
    select type 1 - EFI Label
  > quit

  Repeat above steps with c7t3d0.

  Should now have a partition tables that supports 9TB.

  Apply partition configuration to create 400GB slices:
  % fmthard -s vtoc.out /dev/rdsk/c7t2d0s2
  fmthard:  New volume table of contents now in place.
  % fmthard -s vtoc.out /dev/rdsk/c7t3d0s2
  fmthard:  New volume table of contents now in place.

  Format should report the following:

  Part      Tag    Flag     First Sector           Size           Last Sector
  0        usr    wm                34        400.00GB            838860833
  1        usr    wm         838860834        400.00GB            1677721633
  2 unassigned    wm                 0             0                 0
  3 unassigned    wm                 0             0                 0
  4        usr    wm        1677721634          1.70TB            5328443937
  5        usr    wm        5328546338          6.61TB            19521458142
  6        usr    wm        5328443938         50.00MB            5328546337
  8   reserved    wm       19521458143          8.00MB            19521474526

  Slices 0,1 will be the log devices  Slices 4 - 6 are extra space.

################################################################################
Create 2 mirrored devices of size 400GB each.
################################################################################
  Create the metadevice state database on the first 2 logical devices.
  % metadb -a -f c7t0d0s6 c7t1d0s6

  Create the metadevices using first 2 slices on each lun:
  % metainit d211 1 2 c7t2d0s0 c7t2d0s1 -i 256b
  % metainit d212 1 2 c7t3d0s0 c7t3d0s1 -i 256b

  Mirror the 2 metadevices.
  % metainit d210 -m d211 d212

  Create the soft partitions:
  % metainit d200 -p d210 838860800b
  % metainit d201 -p d210 409599M

  Verify new metadevices and soft partitions created:
  % metastat

################################################################################
Create target groups for exporting COMSTAR luns.  Create 1 target group
per QLT port.  
################################################################################
  To create/destroy target groups, stmf (scsi target mode framework) must
  be disabled if enabled:
  % svcadm disable stmf

  Create target groups
  % stmfadm create-tg tg0
  % stmfadm create-tg tg1

  Verify target groups created:
  % stmfadm list-tg

  Display target WWN ports
  % stmfadm list-target
  Target: wwn.2101001B32B20xxx
  Target: wwn.2100001B32920yyy

  Assign 1 QLT port per target group
  % stmfadm add-tg-member -g tg0 wwn.2101001B32B20xxx
  % stmfadm add-tg-member -g tg1 wwn.2101001B32B20xxx
  % stmfadm add-tg-member -g tg0 wwn.2100001B32920yyy
  % stmfadm add-tg-member -g tg1 wwn.2100001B32920yyy

  Verify WWN ports added to target groups
  % stmfadm list-tg -v

  Re-enable stmf service
  % svcadm enable stmf

################################################################################
Incorporate Metadevices d200-201 into COMSTAR, add in LUN naming scheme, and
add views.
################################################################################

  In the naming scheme, "C" in the 7th to last octet was used to denote a
  redo node lun. The "x" is used to represent one of the 4 Sun Fire
  X4270 M2 systems. The Last 4 octets are the metadevice numbers.  

  GUID string format: 600144F0A3770E00000000000C0xD200

  The following add logical units for the first of 4 systems: 
    % stmfadm create-lu -p guid=600144F0A3770E00000000000C01D200 -p wcd=true /dev/md/rdsk/d200
    % stmfadm create-lu -p guid=600144F0A3770E00000000000C01D201 -p wcd=true /dev/md/rdsk/d201

  Now add in COMSTAR views:
    % stmfadm add-view -t tg0 600144F0A3770E00000000000C01D200
    % stmfadm add-view -t tg0 600144F0A3770E00000000000C01D201
    % stmfadm add-view -t tg1 600144F0A3770E00000000000C01D200
    % stmfadm add-view -t tg1 600144F0A3770E00000000000C01D201

  Verify that the views have been created:
   % stmfadm list-lu -v

################################################################################
Initiate device discovery on the database host for LOG devices
################################################################################
 
  Verify that the luns are now visible on the database host using
  the fcinfo(1M) utility after making physical cable connections
  between the HBA on the Sun Fire X4270 M2 and the HBA on the
  database host with the Storage Area Network switch.

  % fcinfo hba-port | grep "^HBA" | awk '{print $4}'|while read ln
  > do
  >	echo Local Port WWN: $ln
  >	fcinfo remote-port -p $ln -s
  > done

################################################################################
Label luns on the database host.
################################################################################

  On the database host, use the format(1M) utility to label
  the new luns.

