vanheusden.com

contents

• what is it
• iSCSI on a microcontroller?!
• why
• video of it running
• pictures of it talking to e.g. Linux and AIX
• make one yourself
• performance
  • 1 SD card on an ESP32
  • 4 SD cards on 4 ESP32s in a RAID-5 setup
  • ESP32 with an Ethernet interface
  • Teensy 4.1

what is it

iESP is an iSCSI target for microcontrollers. Initially it was written for the Wemos32 but other boards followed.

It can serve a disk image file on an SD-card as if it was a virtual disk. That way you can e.g. boot a VMware cluster from an SD-card.

iSCSI SANs are usually large boxes with tons of harddisks/SSDs consuming loads of power (e.g. this), an ESP32 not so much.

what is so special about it?

The iSCSI protocol is (imho) quite an endeavour to implement, especially on a memory-limited system as the ESP32. During development I noticed that there are a few differences between 'initiators' in how strict they adhere to the protocol as described in the RFC. Also FreeBSD (14.0) gives all kinds of usefull debug-logging, Linux somewhat and AIX and Windows let you guess what goes wrong - I eventually succeeded by comparing network traces (using wireshark) between my esp32 target and 'tgt'.

but why?!

Because I can.

video

YouTube

pictures

hardware

Linux system connecting to iESP

FreeBSD (14) system connecting to iESP

IBM AIX 7.2 connecting to iESP

I followed the steps from this website.
I also had to invoke (extra step): chdev -l hdisk1 -a reserve_policy=no_reserve -U or else AIX will invoke the 'RESERVE' SCSI command which are not implemented in iESP.

Microsoft Windows 10 (home edition) connecting to iESP

VMWare 6.0.0 connecting to iESP

make one yourself

All code is on GitHub (together with instructions). License is the MIT license.

hardware setup

You only need an SD-card reader, connect it to the SPI pins of the ESP32 (SCK 14, MISO 2, MOSI 15 and CS on pin 12). I used this reader from Adafruit (this one works as well) and a wemos32.

You can connect a red LED to GPIO 32, green LED on GPIO 17 and yellow LED GPIO 16. Green blinks for read access, yellow for write access, red for errors/warnings. Green and yellow are both on during WiFi setup.

Lastly you can connect a TM1637 display to pin 21 and 22. It will then show startup-codes, like real hardware does :-)

performance

The performance depends on the speed of WiFi and the SD-card. A quick test (software version a19934d) with a random (SDHC) 32GB card shows 418kB/s linear read and 79 iops for random reads.

click on an image to see it in full size

ESP32 & an SD card

Block size versus latency:

4 node RAID-5 setup

Here I configured 4 Wemos32 to run iESP. Each had a 32 GB SD card, all connected to the same WiFi. My laptop then connected to each iSCSI target and combined them in one RAID-5 device on Linux. I then created the following graphs using the included "block-speed.py" script.

Photo of the configuration:

For what it is worth; a plot of the WiFi spectrum at time of the performance benchmark:

Analysis with iometer (measurer on the Linux laptop, GUI on an other (windows-)laptop):

Ethernet

Some ESPs have an Ethernet adapter on board. Like the wt32-eth01. With that board I ran my own my own benchmark which produces the following graph (using gnuplot):

CPU load & I/O wait during a run of block-speed.py against iESP over Ethernet (retrieved via SNMP from the device itself):

Teensy 4.1

The Teensy 4.1 microcontroller by Paul Stoffregen is a 600 MHz ARM based Arduino compatible microcontroller. It has an on-board SD-card reader and with some soldering an Ethernet interface. The tight coupling of all components gives speeds (with iESP or iTEENSY) of up to 6 MB while accessing the SD-card.

Bandwidth of the Teensy 4.1 compared to the ESP in different configurations:

Same benchmark data but a plot of the IOPs metric:

Here you see that this microcontroller easily maxes out the 100Mb Ethernet adapter it has on board:

screen output

This output is visibile in the serial console of the microcontroller:

...
19317] PDU/s: 0.39, send: 0.04 kB/s, recv: 0.10 kB/s, written: 0.00 kB/s, read: 0.00 kB/s, syncs: 0.00/s, unmaps: 0.00/s, load: 0.03%, mem: 110580
24437] PDU/s: 0.20, send: 0.01 kB/s, recv: 0.01 kB/s, written: 0.00 kB/s, read: 0.00 kB/s, syncs: 0.00/s, unmaps: 0.00/s, load: 0.01%, mem: 110580
29557] PDU/s: 0.20, send: 0.01 kB/s, recv: 0.01 kB/s, written: 0.00 kB/s, read: 0.00 kB/s, syncs: 0.00/s, unmaps: 0.00/s, load: 0.01%, mem: 110580

miscellaneous