Getting started with iSCSI part I-Openfiler Fundamentals

After having read a number of articles detailing the use of Openfiler and Real Application Clusters I was intrigued, especially by the prospect to set up my own extended distance cluster. There was an excellent article by vnull (Jakub Wartak) on OTN about how to do this. So Openfiler seems to be my “SAN”, provided by a Xen based domU.

UPDATE 221219: as you can imagine, based on the description above this article is now completely out of date and should no longer be referred to.

So-what is that Openfiler thing after all? The project website ( states:

  • “Openfiler is a powerful, intuitive browser-based network storage software distribution” and
  • “Openfiler delivers file-based Network Attached Storage and block-based Storage Area Networking in a single framework”
  • And above all it’s free!

Its closest free competitor known to me might be FreeNAS, which is not based on Linux (some BSD dialect, without disrespect)
Openfiler is build on the rPath distribution, the iSCSI component is based on Cisco’s open iscsi implementation. I am primarily interested in the iSCSI implementation, but the appliance offers a lot more, in two categories:

File based networking support

  • Network File System
  • Server Message Block/Common Internet File System
  • File Transfer Protocol

And it also offers directory support, too!

  • Lightweight Directory Access Protocol
  • Network Information Service
  • Active Directory support…

… All of which I didn’t test (yet), maybe later, when the day has 48 hours and Santa Claus come through our chimney. So rather unlikely right now…

Like all new software I get it has to be virtualisable, especially in Oracle VM which I am running on my big iron box. By the way-does anyone know an ATX mainboard that can take more than 8 GB of memory? I would so love to get the MacPro with a Nehalem and 16G of memory but it’s a tad on the expensive side. Buy I digress.

Installing Openfiler in VMWare (Server) which I am running on my slower performing hardware was quite simple: make the installation image available to it, start and follow the steps on the screen. Honestly it’s as simple as that.

For Xen 3.1 (Oracle VM), I had to go through a lot more trouble. Strangely there is no xen install documentation even though a tar.gz is available for download. Start by downloading that from the website, then create a domU:

  • Create a root volume for the domU (5GB) and format it.Easiest done using lvcreate if you use LVM, but a dd if=/dev/zero of=root_disk bs=1G count=5 does the trick as well. Next format that logical volume with ext3.
  • Mount the root volume and extract the image on to it, using the loopback driver.
  • Label the file system as “root”, using e2label
  • Copy the Openfiler kernel and ramdisk to your dom0You need to do this or otherwise the Openfiler appliance won’t start (no use of pygrub here!). The example assumes you are copying it to /openfiler/ as directory.
  • Use a custom configuration file such as this one:
name = "openfiler"
 uuid = " 203e2874-b08c-4066-7166-cada1b5b7341"
 maxmem = 256
 memory = 128
 vcpus = 1
 kernel= "/openfiler/vmlinuz-"
 root=" /dev/xvda1 ro"
 on_poweroff = "destroy"
 on_reboot = "restart"
 on_crash = " restart"
 disk = [ "phy:/dev/openfiler_vg/openfiler_root_lv,xvda1,w", … ]
 vif = [ "mac=00:16:3e:38:75:88,bridge=xenbr0" ]

After startup, use ifconfig in the console to assign an IP and you are done. More detailed step by step instructions can be found here:

Openfiler administration

Although your favourite Linux distribution might come with the same packages as Openfiler, the advantage of the latter is to have all the administration through a common GUI, presented through a web server which starts as part of the boot process.  I have to admit however that yast2 is a pretty decent tool but still requires you to know more about what you are about to do.

Start by pointing your browser to https://hostname:446 to access the GUI, if you are using Firefox then you need to add a security exception.

To access all the great functionality, log in as user “openfiler”, with initial password “password”. A good idea is to change the password now!

iSCSI 101

Greatly simplified, Internet SCSI works by encapsulating and transporting block level data over TCP/IP. The big driver is the “reuse your existing infrastructure” rather than investing in a storage area network. Instead of Fibre Channel, TCP/IP is the transport layer.

And therein lies the problem: TCP/IP has been developed to allow communication over potentially unreliable networks. In a crisis situation, a number of routers could have failed, and TCP-based communication would try and find alternative ways to deliver packets. For a storage specialist, that sounds like overhead and indeed, pure software iSCSI is quite expensive in terms of CPU. Some storage vendors came up with TCP Offload Engines (ToE) and iSCSI HBAs which perform some of the work necessary to encapsulate payload into a TCP frame.

But I am not aiming at line rate throughput, all I need is a simple solution to act as a storage area network-software iSCSI is just fine.

On the “client” side, you need a so called iSCSI initiator to access the iSCSI target. In this respect terminology is similar to parallel SCSI.

The next article will delve into more detail about creating iSCSI targets which the RAC nodes then mount.