New IT forum
12 August 2022, 02:32:43 pm *
Welcome, %1$s. Please login or register.

: CubieBoard 2 and Cubietruck both now in stock.
 
Pages: [1]

Author Topic: 150 M/s NFS, 50% I/O Delays! eSATA Sheeva Multi  (Read 4840 times)

JustaGuy

  • Newbie
  • *
  • Posts: 38
    • My (temporary one-page) website
150 M/s NFS, 50% I/O Delays! eSATA Sheeva Multi
« on: 09 August 2010, 09:51:56 pm »

I have currently the Sheeva set to provide the attached eSATA Drobo-S as an NFS share, and am having atroious throughput issues.

I wanted to change it over to iSCSI to see if that would help, but got bogged down in the kernel upgrade part of the process, and pretty much gave up on that until I have the time to fumble through the learning curve associated with it- which could possibly be never.

So now I have NFS already going, and there's occasional 150 M/s (read & write) in iotop on the client side- but occasional.

The client's breezing through the vmtar operation with it's 4x 7200rpm SAS RAID10, 32GB RAM @ 667mHz, 2x Quad Xeon 5470 3.3Ghz.
The client side reports consistent I/O delays between 20% - 65%.

On the sheeva, htop shows there being 8 nfsd threads, 61 - 65 / 501 MB RAM and CPU stays between 45% - 100%, usually 100%.

I usually like to have the vzdumps at 200000 K/s bandwidth limit so they can blast through & get done, but now unless I tell them to use less than 15000 K/s, I get timeouts on any operation that tries to make use of the Sheeva's NFS share.

The network's all wired Cat6, all Gigabit.
Right now the switch doesn't do jumbo frames. There's another that can, but I haven't gotten around to installing it yet.
It seems to me the switch being at 1500 mtu isn't the bottleneck right now, so I likely won't upgrade the switch until the rest of the SAN is up to speed.
There are other factors delaying the network upgrade too, but not relevant to this segment except for the switch mtu.

This seems a long shot, but I'll present the idea anyhow.
It crossed my mind to configure a balance-alb bond with the RJ45 & WiFi, but it's the 1500 mtu router that has Wireless-N.
I'd rather be rid of it eventually- unless the bond would do some good, then I'd implement it now & change out the router for an access point just for the Sheevas class of Wireless suffix (G, N) later.
I don't know if WiFi can do jumbo frames, but I assume not.

I swapped the SD card with one double the size, that's a class 4 rather than the class 6 one it came with and I don't see much difference in performance.
I haven't done formal testing between the 2- just what I gather from watching iotop on the client & htop on the Sheeva.

Can the memory be increased?

The USB port's shot for some reason, so swap's out of the question.

What do people think?
Insights & feedback's totally welcome.
« Last Edit: 09 August 2010, 09:57:21 pm by JustaGuy »
Logged

Sakis3G

  • Newbie
  • *
  • Posts: 13
    • Sakis3G blog
Re: 150 M/s NFS, 50% I/O Delays! eSATA Sheeva Multi
« Reply #1 on: 10 August 2010, 07:46:56 am »

So now I have NFS already going, and there's occasional 150 M/s (read & write) in iotop on the client side- but occasional.
Just out of curiosity, I would like to see peaks with two clients.

On the sheeva, htop shows there being 8 nfsd threads, 61 - 65 / 501 MB RAM and CPU stays between 45% - 100%, usually 100%.
Sounds like a bottleneck already. I would try changing eth0 for a while to 100Mbps, and check if CPU usage drops (you need to further decrease vzdumps for this). If you are unable to otherwise alter infrastructure, you may be able to force 100Mbps by using a "4-clones-destroyed" cat5.

I usually like to have the vzdumps at 200000 K/s bandwidth limit so they can blast through & get done, but now unless I tell them to use less than 15000 K/s, I get timeouts on any operation that tries to make use of the Sheeva's NFS share.
Right now seems too much optimistic a 200000K. Even if possible with one client, how about two? Anyway, if it is 15000K, then getting eth0 to 100Mbps means nothing will change: 100Mbps can do 15000K as well. So, once more I don't think it is network which introduces bottleneck.

The network's all wired Cat6, all Gigabit.
Right now the switch doesn't do jumbo frames. There's another that can, but I haven't gotten around to installing it yet.
It seems to me the switch being at 1500 mtu isn't the bottleneck right now, so I likely won't upgrade the switch until the rest of the SAN is up to speed.
There are other factors delaying the network upgrade too, but not relevant to this segment except for the switch mtu.
Jumbo frames could help if high CPU usage on Sheeva is exclusively derived by CPU being weak, and not by involved hardware (Ethernet, SATA) generating interrupts in a rate CPU cannot serve. Thing is that you were previously doing it without Jumbo frames, so it is possible. Going Jumbo just for Sheeva to be able to handle the load, will:
  • Force you to make sure all systems on same network are able to do Jumbo,
  • Some applications on your network (gaming, VoIP) may not like lag introduced by Jumbo.
I would definitely test going Jumbo (just to check if it helps), but I think I wouldn't change whole infrastructure because of Sheeva (ROI-wise). Better of throw it way (for the time being) if it needs Jumbo.

This seems a long shot, but I'll present the idea anyhow.
It crossed my mind to configure a balance-alb bond with the RJ45 & WiFi, but it's the 1500 mtu router that has Wireless-N.
I'd rather be rid of it eventually- unless the bond would do some good, then I'd implement it now & change out the router for an access point just for the Sheevas class of Wireless suffix (G, N) later.
I don't know if WiFi can do jumbo frames, but I assume not.
I think that bonding yet another interface would introduce even more IRQs which Sheeva seems unable to handle. Going through this would make sense only if you had discovered Sheeva being able to serve 5 clients with vzdumps @ 200000K while being at most at 80% CPU usage. But network is not the bottleneck right now.

I swapped the SD card with one double the size, that's a class 4 rather than the class 6 one it came with and I don't see much difference in performance.
It shouldn't make any difference, unless you were swapping on it.

The USB port's shot for some reason, so swap's out of the question.
You mean its broken?

How does Sheeva behaves if you locally copy files to/from esata? Does it still do 150 M/s?

Sakis
Logged

JustaGuy

  • Newbie
  • *
  • Posts: 38
    • My (temporary one-page) website
Re: 150 M/s NFS, 50% I/O Delays! eSATA Sheeva Multi
« Reply #2 on: 10 August 2010, 11:30:41 pm »

So now I have NFS already going, and there's occasional 150 M/s (read & write) in iotop on the client side- but occasional.
Just out of curiosity, I would like to see peaks with two clients.

I won't be able to test this with 2 of the same for quite awhile, there are plans for additional nodes, but it's a low proirity now- maybe next summer if all goes well.

I'd have to build a small test box just for the purpose of generating NFS traffic.

On the sheeva, htop shows there being 8 nfsd threads, 61 - 65 / 501 MB RAM and CPU stays between 45% - 100%, usually 100%.
Sounds like a bottleneck already. I would try changing eth0 for a while to 100Mbps, and check if CPU usage drops (you need to further decrease vzdumps for this). If you are unable to otherwise alter infrastructure, you may be able to force 100Mbps by using a "4-clones-destroyed" cat5.

I'll give that a try.
Will have to look into it some since I don't know off the top of my head how that would be done.
I'm imagining there's a setting I can do in /etc/network/interfaces similar to the one in Windows where I can specify the 'Negotiation Speed' of the interface, eg. 'Auto' '1G Auto' '10/100 Full Duplex' etc...


I usually like to have the vzdumps at 200000 K/s bandwidth limit so they can blast through & get done, but now unless I tell them to use less than 15000 K/s, I get timeouts on any operation that tries to make use of the Sheeva's NFS share.
Right now seems too much optimistic a 200000K. Even if possible with one client, how about two? Anyway, if it is 15000K, then getting eth0 to 100Mbps means nothing will change: 100Mbps can do 15000K as well. So, once more I don't think it is network which introduces bottleneck.

I put the value up that high to essentially remove whatever default limits are defined in some other config file somewhere.
I didn't measure, but it seemed to reduce backup completion times significantly, from hours to minutes.

The network's all wired Cat6, all Gigabit.
Right now the switch doesn't do jumbo frames. There's another that can, but I haven't gotten around to installing it yet.
It seems to me the switch being at 1500 mtu isn't the bottleneck right now, so I likely won't upgrade the switch until the rest of the SAN is up to speed.
There are other factors delaying the network upgrade too, but not relevant to this segment except for the switch mtu.
Jumbo frames could help if high CPU usage on Sheeva is exclusively derived by CPU being weak, and not by involved hardware (Ethernet, SATA) generating interrupts in a rate CPU cannot serve. Thing is that you were previously doing it without Jumbo frames, so it is possible. Going Jumbo just for Sheeva to be able to handle the load, will:
  • Force you to make sure all systems on same network are able to do Jumbo,
  • Some applications on your network (gaming, VoIP) may not like lag introduced by Jumbo.
I would definitely test going Jumbo (just to check if it helps), but I think I wouldn't change whole infrastructure because of Sheeva (ROI-wise). Better of throw it way (for the time being) if it needs Jumbo.

One of my goals with the network is for it to be segmented to allow for that.
I don't have the details sorted yet, but there are 2 Sheevas that can each have allocated an exclusive GB interface capable of TCP &/or iSCSI offload, to bring them into one of the nodes hosting virtual machines.
So far I've figured out I can set seperate bridges on these interfaces to 0.0.0.0 & essentially have a pass-through to a virtualized firewall managing things such as multi-wan, but it's nowhere near done yet & there's ample room for change.
--
According to another thread:
http://www.newit.co.uk/forum/index.php/topic,525.0.html
where I asked about the feasibility of doing so, it seems possible so I'll add that to the to-do list.

This seems a long shot, but I'll present the idea anyhow.
It crossed my mind to configure a balance-alb bond with the RJ45 & WiFi, but it's the 1500 mtu router that has Wireless-N.
I'd rather be rid of it eventually- unless the bond would do some good, then I'd implement it now & change out the router for an access point just for the Sheevas class of Wireless suffix (G, N) later.
I don't know if WiFi can do jumbo frames, but I assume not.
I think that bonding yet another interface would introduce even more IRQs which Sheeva seems unable to handle. Going through this would make sense only if you had discovered Sheeva being able to serve 5 clients with vzdumps @ 200000K while being at most at 80% CPU usage. But network is not the bottleneck right now.

Good to know, thanks.

I swapped the SD card with one double the size, that's a class 4 rather than the class 6 one it came with and I don't see much difference in performance.
It shouldn't make any difference, unless you were swapping on it.

I'd prefer not to have to deal with swap at all- last time I tried on USB, there ensued increasing errors until I wound up having to send the plug back for re-flash.

The USB port's shot for some reason, so swap's out of the question.
You mean its broken?

It surely seems that way, I discovered that while learning how to clone the SD card, which eventually had to happen via the eSATA drive instead.
The process of reaching that conclusion is detailed in this thread, starting from the 12th post:
http://www.newit.co.uk/forum/index.php/topic,511.0.html
Still no reply to either of a couple emails to the manufacturer on that.

How does Sheeva behaves if you locally copy files to/from esata? Does it still do 150 M/s?
I don't know how to see that at this point, the Sheeva's kernel as shipped doesn't support the use of iotop.
Logged
Pages: [1]
 
 

Powered by MySQL Powered by PHP SMF 2.0.10 | SMF © 2015, Simple Machines Valid XHTML 1.0! Valid CSS!