914club.com BBS is slow, anyone else notice this? Options

[Qarl]

I notice that the traceroute is taking a different route across the country. Last time I provided a traceroute (about a 4-6 weeks ago, on another thread), there was a different router somewhere that we thought was causing all the hangups.

Oh... and one other point brought up a couple of posts back, and I think this is important in helping diagnose issues...

I too can load the 914club home page really quickly... which supports that fact that a ping or traceroute is going to respond quickly.

It's the loading of the technical BBS.... err I mean, forum that crawls. And I know this is where all the dynamic mumbo jumbo crazy magic stuff happens...

So to repeat myself, and several others... accessing the homepage is quick, dynamic pages is worse, dynamic pages at peak times is worser.... accessing pages and replying with 100 users is the mostest worstest.

[Qarl]

Oh shit, I almost forgot...

:finger2: :finger2:

[lapuwali]

(IMG:style_emoticons/default/biggrin.gif)

OK, that vmstat dump (thanks, Mark), shows:

there aren't many processes waiting to do anything (first column is mostly 0)
the CPU is reasonably busy (20-50% idle)
it's swapping some (mostly single digit numbers in si so, but some double digits)
relatively little disk activity (bi bo numbers, anything under 400 regularly is low)

We're not really correlating this to board "slowness" (seems fine to me right now), but I'm sticking by my guns on this. If one of the admins can repeat the vmstat exactly when the board is definitely running slowly, we'll see what it's like then. If it regularly goes into double digits on si/so, it's hurting. Disk and CPU don't appear to be a bottleneck, at least from what I've seen so far. The fact that the CPU isn't idle yet it's not running user processes means it's busy doing housekeeping tasks (like swapping).

Oh, and :finger2:

[Brad Roberts]

do you really think brad is currently working on fixing the server?
that's a good one ...

If I could physically get to the server... the ram would already be installed. I believed Mark the first time..

I also believe EVERY single person on this BBS with slow response times (not PING times). Pings dont tell us a damn thing about how the database is responding to a query. Since the front page doesnt query the DB like EVERY single post does... it doesnt tax the system.

Now. Can we schedule to get me in there on Tuesday (or this weekend ? so I CAN install some RAM ?


B

[vortrex]

it's not a network issue. you can see you get a fast reponse when setting up the tcp connection with your http request, but a long delay waiting for content ("waiting for www.914world.com"). when the content is ready, it loads quickly. if it were packet loss, congested network, etc you would get a slow loading page more so than a slowing responding page. someone should run tcpdump on the server and watch the connections being made, might give some more clues.

[mercdev]

I don't work with Apache, but does it use bandwidth throttling? Seems I remember similar circumstances with a *nix box on a network at Sprint (the box was on a 1GB ethernet switch that connected to a couple OC3's). Engineers sat around scratching their heads and throwing parts at it (upgraded CPU/more ram, new NICs, etc) until they found that "someone" had set the max throughput per session to some unrealistic setting in an effort to "tune" the server.

Do you have any large files people could download from the server to see what the avg. KB/S throughput they're getting is? Most broadband connections get 400-500 (or higher) where something in the 130-150 range usually indicates a saturated T1 or some time of restriction taking place.

(Not bitching at all, I love this site!)

[SirAndy]

Now. Can we schedule to get me in there on Tuesday (or this weekend ? so I CAN install some RAM ?

i'm ready to switch over to the compaq box, i only use it for my personal website these days, all my eCommerce stuff has been moved to other (newer) servers ...

4GB RAM, dual CPU's, raid-array with lots-o-disk space, etc. etc. etc.

should only take me a afternoon to switch everything over and kill the linux box.
(IMG:style_emoticons/default/smash.gif) Andy

[Gint]

You guys are something else... (IMG:style_emoticons/default/wacko.gif)

You're all correct to varying degrees. There are network bottlenecks, peak usage times for the internet as a whole, high db access, and a lot of swapping going on. All of these things together cause slow downs at peak periods.

I created a shell script that runs date, uptime, and a vmstat at 5 second intervals for 12 interations. It's been cron'd to run 33 minutes after every hour. We'll see what happens over the next full day or so. Of course the acid test would be Tuesday.

Right now (pretty quiet and the board is fairly speedy):

Fri Sep 3 22:33:00 PDT 2004

22:33:00 up 6:07, 1 user, load average: 0.69, 0.46, 0.45

procs memory swap io system cpu
r b w swpd free inact active si so bi bo in cs us sy id
3 0 1 94932 5232 4488 180880 4 20 118 156 206 115 36 8 56
0 0 0 94932 5260 4488 180308 0 0 0 171 141 97 10 2 88
0 1 1 94932 4936 4492 182068 0 0 2 70 184 107 16 3 82
2 0 0 95124 3508 5240 182324 0 38 0 434 255 227 54 15 31
1 0 0 95124 4220 4792 181692 0 0 0 425 201 102 53 11 36
0 0 0 95124 4244 4784 181736 0 0 0 42 133 61 6 1 94
0 0 0 95124 3812 6304 181952 0 0 0 33 131 80 59 16 25
0 0 0 95124 3932 4784 182040 0 0 0 444 137 50 15 2 83
1 1 0 95124 3908 4784 183088 0 0 0 18 114 58 5 1 94
0 0 0 95708 3748 4660 184436 0 117 5 157 157 119 23 4 73

[Gint]

Here's this hour's results (even quieter then the previous hour):

Fri Sep 3 23:33:01 PDT 2004

23:33:01 up 7:07, 1 user, load average: 0.44, 0.56, 0.53

procs memory swap io system cpu
r b w swpd free inact active si so bi bo in cs us sy id
2 1 1 95004 12148 4092 184604 4 18 103 152 198 110 34 8 58
0 0 0 95004 11496 4084 185080 0 0 0 135 156 111 31 4 65
0 0 0 95004 11240 5496 184264 0 0 0 349 218 76 53 14 33
0 0 0 95004 11620 4084 184176 0 0 0 21 145 66 18 3 79
0 0 0 95004 11620 4084 184200 0 0 3 38 172 80 1 1 99
0 0 0 95004 11620 4084 184724 0 0 0 27 126 62 8 2 90
0 0 0 95004 11620 4084 184404 0 0 0 26 186 75 4 1 94
0 0 0 95004 11628 4084 184420 0 0 4 56 205 112 14 3 83
0 0 0 95004 11628 4084 184424 0 0 0 10 116 43 0 0 100
0 0 0 95004 11620 4084 184848 0 0 0 31 138 69 9 1 90


Users at this hour:
7 guests, 19 members 2 Anonymous Members
I'll post more in the morning. The results up until then should provide a nice quiet baseline. We'll be able to see it go up as the morning goes on. I could script a traceroute via ssh to my mail server so I could include it in the log file for the vmstat script output, but the ROI for that work ain't worth it.


Here's a traceroute from my mail server. It's interesting to note that there is a lag of 50 seconds between hop 7 and 8. The response times don't show it, but it's there. I don't know what it means since it obviously doesn't take that long to contact the server. Anywho...

> traceroute 914world.com
traceroute to 914world.com (66.250.97.205), 64 hops max, 44 byte packets
1 access01-fe6-0-18.ftc.frii.net (216.17.222.6) 0.448 ms 0.422 ms 0.296 ms
2 core01-fe6-0-701.ftc.frii.net (216.17.230.17) 0.749 ms 1.019 ms 0.928 ms
3 core01-atm3-0-32.den.frii.net (216.17.230.42) 3.698 ms 3.817 ms 4.784 ms
4 f29.ba01.b006467-1.den01.atlas.cogentco.com (66.250.5.253) 3.879 ms 3.803 ms 3.805 ms
5 g9-2.core01.den01.atlas.cogentco.com (66.28.5.21) 3.927 ms 3.829 ms 3.808 ms
6 p4-0.core02.sfo01.atlas.cogentco.com (66.28.4.130) 238.887 ms 205.652 ms 218.214 ms
7 g50.ba01.b003070-1.sfo01.atlas.cogentco.com (66.28.5.182) 28.611 ms 27.725 ms 28.216 ms
8 209.17.64.166 (209.17.64.166) 28.969 ms 29.106 ms 29.114 ms
9 64.237.0.250 (64.237.0.250) 29.632 ms 29.475 ms 29.830 ms
10 mail.914world.com (66.250.97.205) 29.281 ms 29.258 ms 29.638 ms


> ping 914world.com
PING 914world.com (66.250.97.205): 56 data bytes
64 bytes from 66.250.97.205: icmp_seq=0 ttl=48 time=29.748 ms
64 bytes from 66.250.97.205: icmp_seq=1 ttl=48 time=29.098 ms
64 bytes from 66.250.97.205: icmp_seq=2 ttl=48 time=29.645 ms
64 bytes from 66.250.97.205: icmp_seq=3 ttl=48 time=29.377 ms
64 bytes from 66.250.97.205: icmp_seq=4 ttl=48 time=29.523 ms
64 bytes from 66.250.97.205: icmp_seq=5 ttl=48 time=29.124 ms
64 bytes from 66.250.97.205: icmp_seq=6 ttl=48 time=29.343 ms
64 bytes from 66.250.97.205: icmp_seq=7 ttl=48 time=29.070 ms

[Gint]

Deader than a doornail now:

4 guests, 13 members 3 Anonymous Members


Sat Sep 4 00:33:00 PDT 2004

00:33:00 up 8:07, 1 user, load average: 0.21, 0.22, 0.27

procs memory swap io system cpu
r b w swpd free inact active si so bi bo in cs us sy id
2 0 0 99384 4152 4092 189356 3 17 90 143 191 105 32 7 60
0 0 0 99384 4348 4064 188552 0 2 10 116 126 87 1 1 98
0 0 0 99384 4368 4064 188560 0 0 0 15 113 46 0 0 100
0 0 0 99384 4016 4200 188524 0 45 0 242 134 62 65 16 18
0 0 0 99384 4336 4192 188068 0 0 0 37 148 64 10 1 89
0 0 0 99384 4308 4192 187892 1 2 2 15 165 82 7 1 92
0 0 0 99384 4316 4192 188020 0 0 2 38 156 52 0 1 99
1 0 0 99384 4316 4192 188020 0 0 0 10 117 45 1 0 99
0 0 0 99384 4204 4720 188020 0 0 0 184 133 58 62 19 19
1 0 0 99384 4188 4716 188036 0 0 0 19 117 58 4 0 96
0 0 0 99388 4020 4720 187792 0 40 0 71 138 75 15 2 83
0 0 0 99388 4020 4716 187792 0 0 0 25 118 46 0 1 99

/usr/sbin/traceroute www.914world.com
1 access01-fe6-0-18.ftc.frii.net (216.17.222.6) 0.530 ms 0.449 ms 0.415 ms
2 core01-fe6-0-701.ftc.frii.net (216.17.230.17) 1.374 ms 0.557 ms 0.646 ms
3 core01-atm3-0-32.den.frii.net (216.17.230.42) 4.273 ms 3.582 ms 4.074 ms
4 f29.ba01.b006467-1.den01.atlas.cogentco.com (66.250.5.253) 3.668 ms 3.909 ms 3.750 ms
5 g9-2.core01.den01.atlas.cogentco.com (66.28.5.21) 4.034 ms 4.133 ms 4.884 ms
6 p4-0.core02.sfo01.atlas.cogentco.com (66.28.4.130) 27.587 ms 28.403 ms 28.783 ms
7 g50.ba01.b003070-1.sfo01.atlas.cogentco.com (66.28.5.182) 28.071 ms 28.048 ms 27.662 ms
8 209.17.64.166 (209.17.64.166) 29.317 ms 29.904 ms 30.597 ms
9 64.237.0.250 (64.237.0.250) 29.540 ms 29.564 ms 29.594 ms
10 914world.com (66.250.97.205) 29.850 ms 29.430 ms 29.588 ms

> ping www.914world.com
PING www.914world.com (66.250.97.205): 56 data bytes
64 bytes from 66.250.97.205: icmp_seq=0 ttl=48 time=29.776 ms
64 bytes from 66.250.97.205: icmp_seq=1 ttl=48 time=29.579 ms
64 bytes from 66.250.97.205: icmp_seq=2 ttl=48 time=29.306 ms
64 bytes from 66.250.97.205: icmp_seq=3 ttl=48 time=30.819 ms
64 bytes from 66.250.97.205: icmp_seq=4 ttl=48 time=29.729 ms
64 bytes from 66.250.97.205: icmp_seq=5 ttl=48 time=29.264 ms

[Gint]

Really, really, really quiet now.

2 guests, 7 members 1 Anonymous Members

Sat Sep 4 01:32:59 PDT 2004

01:32:59 up 9:07, 1 user, load average: 0.12, 0.15, 0.10

procs memory swap io system cpu
r b w swpd free inact active si so bi bo in cs us sy id
3 0 0 99676 4560 4340 188828 3 15 81 133 183 100 30 7 64
0 0 0 99832 4348 4336 187952 0 39 0 209 136 85 1 1 98
0 0 0 99832 4348 4336 187952 0 0 0 10 129 49 0 0 100
0 0 0 99832 4348 4336 187952 0 0 0 10 128 46 0 0 100
0 0 0 99832 4348 4336 187960 0 0 1 7 131 53 0 0 100
0 0 0 99832 4348 4336 187964 0 0 0 14 129 53 0 0 100
0 0 0 99832 4348 4828 188116 0 0 0 18 122 59 25 6 69
0 0 0 99832 4244 4336 187972 0 0 0 38 128 47 41 10 49
0 0 0 99832 4260 4336 187972 0 0 0 8 120 62 6 1 93
0 0 0 99832 4256 4336 187972 0 0 0 30 110 43 0 0 100
0 0 0 99832 4184 5588 188124 0 0 0 22 114 57 47 11 42
1 0 0 99832 4172 4544 187724 0 1 0 21 134 46 21 3 76


traceroute www.914world.com
1 access01-fe6-0-18.ftc.frii.net (216.17.222.6) 0.404 ms 0.424 ms 0.457 ms
2 core01-fe6-0-701.ftc.frii.net (216.17.230.17) 0.669 ms 0.557 ms 0.587 ms
3 core01-atm3-0-32.den.frii.net (216.17.230.42) 4.226 ms 4.102 ms 3.595 ms
4 f29.ba01.b006467-1.den01.atlas.cogentco.com (66.250.5.253) 4.106 ms 4.489 ms 4.007 ms
5 g9-2.core01.den01.atlas.cogentco.com (66.28.5.21) 4.314 ms 3.762 ms 3.877 ms
6 p4-0.core02.sfo01.atlas.cogentco.com (66.28.4.130) 27.558 ms 58.140 ms 28.216 ms
7 g50.ba01.b003070-1.sfo01.atlas.cogentco.com (66.28.5.182) 27.871 ms 27.773 ms 28.242 ms
8 209.17.64.166 (209.17.64.166) 29.123 ms 29.019 ms 29.554 ms
9 64.237.0.250 (64.237.0.250) 29.880 ms 29.672 ms 30.002 ms
10 www.914world.com (66.250.97.205) 29.429 ms 30.066 ms 29.213 ms
ping -c 7 www.914world.com
PING www.914world.com (66.250.97.205): 56 data bytes
64 bytes from 66.250.97.205: icmp_seq=0 ttl=48 time=29.910 ms
64 bytes from 66.250.97.205: icmp_seq=1 ttl=48 time=29.596 ms
64 bytes from 66.250.97.205: icmp_seq=2 ttl=48 time=29.345 ms
64 bytes from 66.250.97.205: icmp_seq=3 ttl=48 time=29.355 ms
64 bytes from 66.250.97.205: icmp_seq=4 ttl=48 time=29.073 ms
64 bytes from 66.250.97.205: icmp_seq=5 ttl=48 time=29.923 ms
64 bytes from 66.250.97.205: icmp_seq=6 ttl=48 time=29.339 ms

--- www.914world.com ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max/stddev = 29.073/29.506/29.923/0.295 ms

[lapuwali]

Gint, that lag between hops 7 & 8 on your traceroute is the name lookup for that hop timing out. Use -n to skip the name lookup, and it will sail right past that. The reverse DNS for that 209 hop is having problems. You can't look it up with dig -x, either.

[Gint]

Gint, that lag between hops 7 & 8 on your traceroute is the name lookup for that hop timing out. Use -n to skip the name lookup, and it will sail right past that.

Thanks James. That makes sense now that I'm awake. Done. With as many as 50 users the box has been running pretty well since last night.

8 guests, 50 members 3 Anonymous Members


Sat Sep 4 11:33:01 PDT 2004

11:33:02 up 19:07, 1 user, load average: 0.57, 1.06, 1.16

procs memory swap io system cpu
r b w swpd free inact active si so bi bo in cs us sy id
1 0 0 91396 36304 34716 136200 2 10 83 113 171 91 24 6 70
0 0 0 91396 30680 36692 139496 0 0 0 320 234 131 66 16 18
1 0 0 91396 28464 34724 142488 0 0 0 307 165 79 81 19 0
0 0 0 91396 27016 34736 143732 1 0 2 79 247 138 19 2 79
0 0 0 91240 35180 32884 138912 0 0 0 30 118 78 17 3 80
0 0 0 91240 37444 35172 134184 0 0 0 246 133 85 82 18 0
1 0 0 91240 33284 32892 139764 6 0 20 638 202 137 80 17 3
0 0 0 91240 32544 33832 140988 0 0 0 76 312 144 46 10 44
0 0 0 91240 27156 37040 143164 0 0 0 575 127 67 78 22 0
1 0 0 91240 25476 34968 146460 0 0 0 454 188 101 78 22 0
1 0 0 91240 19804 34996 152360 0 0 339 263 640 206 78 22 0
2 0 0 91240 11980 41532 154816 0 0 1784 63 377 238 47 10 43


Sat Sep 4 12:33:00 MDT 2004

traceroute -n www.914world.com

1 216.17.222.6 0.628 ms 0.317 ms 0.295 ms
2 216.17.230.17 0.992 ms 1.240 ms 0.597 ms
3 216.17.230.42 3.477 ms 3.725 ms 3.367 ms
4 66.250.5.253 4.413 ms 4.888 ms 4.103 ms
5 66.28.5.21 3.975 ms 3.928 ms 3.727 ms
6 66.28.4.130 27.363 ms 28.616 ms 27.560 ms
7 66.28.5.182 27.969 ms 27.883 ms 27.796 ms
8 209.17.64.166 29.784 ms 29.908 ms 29.729 ms
9 64.237.0.250 29.763 ms 29.637 ms 30.197 ms
10 66.250.97.205 30.435 ms 29.652 ms 29.464 ms

ping -c 7 www.914world.com

PING www.914world.com (66.250.97.205): 56 data bytes
64 bytes from 66.250.97.205: icmp_seq=0 ttl=48 time=29.811 ms
64 bytes from 66.250.97.205: icmp_seq=1 ttl=48 time=30.508 ms
64 bytes from 66.250.97.205: icmp_seq=2 ttl=48 time=30.289 ms
64 bytes from 66.250.97.205: icmp_seq=3 ttl=48 time=28.989 ms
64 bytes from 66.250.97.205: icmp_seq=4 ttl=48 time=31.107 ms
64 bytes from 66.250.97.205: icmp_seq=5 ttl=48 time=29.734 ms
64 bytes from 66.250.97.205: icmp_seq=6 ttl=48 time=29.611 ms

--- www.914world.com ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max/stddev = 28.989/30.007/31.107/0.638 ms

[lapuwali]

Interesting. Good data. Note that disk I/O (bi bo) is up substantially (over 600 regularly), and the CPU is, indeed, pegged. Not swapping as much as in earlier dumps.

This is probably entirely moot, since Andy is talking about moving the whole thing to a much better box, but...

My guess is the bottleneck is split between the DB server and Apache competing for CPU time, and a small amount of thrashing on RAM. I'd guess now that just adding RAM wouldn't make a huge difference. The disks are starting to get a bit busy. If sticking with the existing machine was a limitation, I'd next investigate the following:

What's the avg. query rate for MySQL (mysqladmin stat; sleep 5; mysqladmin stat. Subtract the two "Questions" numbers, divide by 5, there's your queries per second.)? What percentage of the HTTP queries are image serving (would require a quick Perl script to parse some access logs)? What's the split between MySQL and Apache in CPU usage (top will tell you this)?

After answering these questions, there are various configuration changes that could be made, most of them "free". However, throwing hardware at the problem is easier, and sounds like it's going to happen, anyway. If I had the luxury, what I'd probably do first is move the MySQL DB to a different box, and leave the site where it is.