Hostmedic Emergency Medicine for Hosting & Server Admins
Subscribe to rss feed

Posts Tagged ‘iscsi’

17
September
No Comments

NFS Mounts on Leopard / MAC OS

mac-linux-nfsNFS mounts are now added using Disk Utility. To add a Linux-hosted NFS share do the following:
  1. Start Disk Utility
  2. Select NFS Mounts from the File menu
  3. Click the plus sign in the lower left corner
  4. Enter in your remote NFS URL info as described
  5. Enter the mount point. If you add it to /Network, then it will show up in the Shared section of Finder, in an entry called All.
  6. If you’re using secure ports on your server, then click the Advanced option and enter -P
  7. Click Verify
  8. Save your settings

for those of you asking about iscsi -

ZFS wasn’t the only AWOL storage technology in Apple’s OS X 10.5 – early builds of Leopard included a built-in iSCSI initiator. When the operating system was finally released in October of 2007, both ZFS and iSCSI were quietly dropped.  Both Microsoft and Linux offer native iSCSI client implementations and Apple is lagging behind here.

In the mean time, those seeking iSCSI connectivity within Mac OS X do have third-party options to consider:

  • Studio Network Solutions offers a free  iSCSI initiator for OS X Tiger and Leopard. The globalSAN iSCSI Initiator is the client-side portion of their SANmp storage sharing solution, and purports to support most of the required protocols. In practice, however, the company cannot support other targets than their own.
12
September
2 Comments

iscsi ≠Speed when it comes to DataBases

Recently a quite large client contacted the HostMedic agency regarding some issues they are having within their network at their datacenter.

They are using VMWare – and have what appears to be an excellent setup – until you step under the hood and tinker for a little while.    This client has done everything “right” – purchasing a SAN from Dell (the axiom line) as well as using Gigabit networking – Their SAN is connected to the network sharing its drives using the iSCSI protocol.

What this client and I discovered the hard way is that it does not matter how fast the disks are – or how many disks you have if your pipe to them is slower than the pipe sticking out of a molasses tree.

I recreated the network locally as best as I could – and then turned on Bonnie. Bonnie is an excellent disk performance utility.   After compiling the latest version, 1.96, (which allows you to measure lag time and even includes a multi-threaded mode) it quickly became apparent that their read times were quite high – interestingly enough higher than their write times (go figure).

The upper limit of “Gigabit Ethernet” is (theoretically) about 125MByte/sec.  Sadly – there is no real way for this client to get more due to their expensive iSCSI device not allowing more than one connection to parallel the data – so of course 125MByte/sec is all you get…   My testing in the real world gave roughly 40mbps.    OUCH -  So copying roughly 250GB of databases would take the better part of 5+ hours to complete.

I ran to google and asked the question: Can you get high throughput with iSCSI with GbE on ESX?

The answer is YES.  But there are some complications, and some configuration steps that are not immediately apparent.

You need to understanding some iSCSI fundamentals, some Link Aggregation fundamentals, and know some ESX internals – none of which are immediately obvious to most folks (including myself).

A few take homes here:

  • Ethernet link aggregation isn’t worth crap in iSCSI environments
  • iSCSI HBA’s don’t buy you much other than boot-from-SAN in ESX,
  • The most common configuration (ESX software iSCSI) is limited to about 100 MB/s per iSCSI target over one-gigabit Ethernet.
  • Adding multiple iSCSI targets adds performance across the board, but configurations will vary according to your array.
  • Maximum per-target performance comes from guest-side software iSCSI, which can make use of multiple Ethernet links to push each array as fast as it can go.
  • accessing iSCSI directly from guest VM’s gives better performance than accessing it via the hypervisor layer.
  • If you are serious about iSCSI in your production environment, it’s valuable to do a bit of investigation learning, and it’s important to do a little engineering during design.    Of course iSCSI is easy to connect and begin using, but like many technologies which excel in terms of their simplicity the default options and parameters may not be robust enough to provide an iSCSI infrastructure which can support your organization.
  • Think about Flow-Control (should be set to receive on switches and transmit on iSCSI targets)
  • Enable spanning tree protocol with either RSTP or portfast enabled
  • Filter / restrict bridge protocol data units on storage network ports
  • Configure jumbo frames (always end-to-end – otherwise you will get fragmented crap)
  • Use Cat6 cables rather than Cat5/5e – the better the cable the better the connection. PERIOD
  • Consider cross-stack Etherchannel trunking for your configuration.
  • Investigate your switch – every switch has its good and bad – such as the amount of port buffers, ram, cpu, etc …

(Strongly) Recommended Additional Reading

  1. Scott Lowe has done an excellent job talking on ESX networking.   Start with his recap here: http://blog.scottlowe.org/2008/12/19/vmware-esx-networking-articles/
  2. Read the vendor’s documentation!

My advice is – stay away from iSCSI for databases – and go with direct storage – iSCSI is not ready for performance-sensitive applications out of the box.   I would suggest that anyone considering iSCSI with VMware should feel confident that their deployments can provide high performance and high availability as long as theyunderstand the “one link max per iSCSI target” ESX iSCSI initiator behavior.

10
September
No Comments

SAN w/o the cost – CentOS + iScsi

Nowadays, the iSCSI technology is quite popular in the storage world

Running iSCSI target on a Fedora system is as easy as “yum install iscsitarget” and configure the thing.

On CentOS – however  iSCSI Enterprise Target (IET) daemon is not in the default Yum repositories – and while CentOS does offer TGT  (Linux SCSI Target Framework) I think for a SAN – IET is best.

In short – this requires us to start from scratch – and build from source.  It is important to note – whenever you update your kernel – you will need to recompile.

The iSCSI Target system

First, some prerequisites :

# yum install openssl-devel kernel-dev gcc rpm-build

Download the latest IET from the Sourceforge repo and put the tgz in /usr/src

# cd /usr/src
# tar xvf iscsitarget-0.4.17.tar.gz
# cd iscsitarget-0.4.17
# make
# make install

Like most everything in linux – now that the make install is complete – your ready to go (except a minor configuration of course).  The iscsi-target init.d script is installed and will be started at boot-time.

Configuring the ietd host is pretty simple and the WIKI has tons of valuable information.
First, lets decide who can connect to the IET daemon :

First you have to have a naming convention for your IQN (iScsi Qualified Name) -   The IQN is an identifier for your iSCSI target.

Ok – so glenn – what the heck is this needed for? – I’m glad you asked..   The iSCSI initiator using the IQN to connect to the disks/lun’s.

# nano /etc/initiators.allow
iqn.namingconvension:mydiskname 10.0.100.0/24

In the above example The subnet 10.0.100.0/24 is allowed to use this iSCSI target.

Next, we’ll create the initiators.deny file, which is pretty straightforward :

 # nano /etc/initiators.deny
ALL:ALL

Time to create the IQN in the ietd configuration file.

# nano /etc/ietd.conf
Target

iqn.namingconvension:mydiskname
IncomingUser username 12345
OutgoingUser username 123456789012
Lun 0 Path=/dev/SAN/diskname,Type=fileio,IOMode=wb
Alias iSCSI for diskname
ImmediateData Yes
MaxConnections 1
InitialR2T Yes

I use the following conventions, as defined in the RFC :

For IncomingUser: Password always 5 characters
For OutgoingUser: Password always 12 characters

I suggest using LVM as a disk backend. The disk can also be /dev/sdb or whatever your heart desires…


The iSCSI Initiator

In Ubuntu/Debian/Fedora/ (or pick your Nix version and place here) there usually is an open-iscsi in the repository. If not, you can always compile it from source at http://www.open-iscsi.org.

I love the CentOS Nix version – so the directions below – are assuming you have a CentOS system you would like to use with your new san.

Let’s install the prerequisites :

# yum install iscsi-initiator-utils
# yum install open-iscsi

Next, define the initiator name. This is in the exact same form as the target name, but it should not be the same. This initiator name is the name (in IQN) of your computer.

# nano /etc/iscsi/initiatorname.iscsi

InitiatorName=iqn.namingconvension:someuniquename

Next, we’re going to configure the authentication and some specials in the iscsid config.

# nano /etc/iscsi/iscsid.conf

node.startup = automatic
node.session.auth.authmethod = CHAP
node.session.auth.username = username
node.session.auth.password = 12345
node.session.auth.username_in = username
node.session.auth.password_in = 123456789012
node.session.timeo.replacement_timeout = 120
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 10
node.conn[0].timeo.noop_out_timeout = 15
node.session.initial_login_retry_max = 10
node.session.cmds_max = 128
node.session.queue_depth = 32
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
node.session.iscsi.FastAbort = No

For more information about these setting, please refer to the open-iscsi page.
Next up, start the thing :

# service iscsi start

Bingo! You just started the iSCSI daemon (all cheer).

Now, we’re going to discover our targets on the target iSCSI host. I assume 10.0.100.1 is the target host in this example.

# iscsiadm -m discovery -t st -p 10.0.100.1
10.0.100.1:3260,1 iqn.namingconvension:mydiskname

As you can see, it found the target we created before. Now, let’s login to it:

# iscsiadm -m node -p 10.0.100.1  -T iqn.namingconvension:mydiskname --login

Congratulations Chap – if you do dh -alh – or dmesg  you will find a new disk inserted (/dev/sdc or whatever).

Mac Data Recovery

Powered by: VineHosting