Possible UserAgent problem

Post #331,549 by drook 8/19/10 2:50:46 PM Reply	Possible UserAgent problem I set up a client site: http://amoderatelife.com If you go there via browser, everything shows up fine. But none of the bots are finding it. I found a test page -- http://www.botsvsbro...lateUserAgent.asp -- that lets you simulate what you get seeing it with various UserAgent strings. It always shows up as 404, no matter what string I use. Anybody have any clues what's going on here? Is something in the WordPress htaccess not working correctly? -- Drew
Post #331,552 by folkert 8/19/10 3:04:45 PM 8/19/10 3:06:04 PM Reply	I am getting a sitemap corruption issue. Everytime I use a browser to get the http://amoderatelife.com/sitemap.xml.gz (after referencing the robots.txt) I can't get any browser to open it properly. And Yes, I know its compressed, but other sites I've seen with compressed Sitemaps display properly. Do you have Compression enabled in apache for this site? Now, I can download the file and open it just fine... which leads me to believe, browsers can't/don't know it compressed. Which also means search engines... won't know either. Edited by folkert Aug. 19, 2010, 03:06:04 PM EDT Expand All History
Post #331,554 by folkert 8/19/10 3:15:49 PM Reply	I've completely imitated GoogleBot MSNBot (and Bing) Yahoo Slurp. No problem seeing content, browsing normally. But the Sitemap things STILL affects everything search and some other pieces like in IE.
Post #331,555 by drook 8/19/10 3:36:35 PM 8/19/10 3:52:41 PM Reply	Thanks, hadn't thought to check the sitemap Strange. I changed the options to not generate it. Regenerated and tried the simulator and it worked for the Firefox UA. Still not for the two bots I'm checking. (Stummbot and Dnsdigger) The old sitemap.xml.gz was still there, so I deleted it. Tried again and none of the UAs are working again. Grrr. === Update: Looks like things were cached. Stumbleupon is working now. (Fingers crossed that it keeps working.) Thanks a ton for the tip. I would never have thought those were related. -- Drew Edited by drook Aug. 19, 2010, 03:52:41 PM EDT Expand All History
Post #331,557 by drook 8/19/10 4:31:08 PM Reply	Hmm, not what I thought I checked several clients. The ones who I moved from Blogger are having problems. When I use the UA simulator page for http://divinehealthfromtheinsideout.com I get a Blogger.com error page saying the site doesn't exist. It hasn't been on Blogger for over two months. Ping and whois both show that it's a Dreamhost account now. Why are some sites still resolving to Blogger, even though DNS was changed months ago? -- Drew
Post #331,562 by folkert 8/19/10 9:01:21 PM Reply	Must be some references into the... Blogger.com sites. I see some blogger.com references in the site code from DHFTIO. But I'm at a loss, since you are using wordpress and well blogger uses something else. Could it be the conversion wasn't as clean as hoped?
Post #331,564 by drook 8/19/10 9:20:05 PM Reply	Those Blogger references look like comment author links -- Drew
Post #331,586 by folkert 8/20/10 5:52:13 AM Reply	Yeah, thought so. But I needed to make sure you saw it that way also.
Post #331,563 by drook 8/19/10 9:17:35 PM Reply	Now it's just toying with me I'm tailing the access log on my site and on the client's site, when I hit them from the UserAgent simulator. On mine I get: 99.155.105.30 - - [19/Aug/2010:17:48:30 -0700] "GET /wp/wp-content/themes/thematic/library/styles/reset.css HTTP/1.1" 304 236 "http://www.botsvsbrowsers.com/SimulatePreview.asp?Method=GET&UserAgent=Mozilla/5.0%20%28X11%3B%20U%3B%20Linux%20i686%3B%20en-US%3B%20rv%3A1.9.2.8%29%20Gecko/20100723%20Ubuntu/10.04%20%28lucid%29%20Firefox/3.6.8%20GTB7.1&url=http%3A//cooklikeyourgrandmother.com/" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.8) Gecko/20100723 Ubuntu/10.04 (lucid) Firefox/3.6.8 GTB7.1 On hers: 99.155.105.30 - - [19/Aug/2010:17:48:11 -0700] "GET / HTTP/1.1" 200 13705 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.8) Gecko/20100723 Ubuntu/10.04 (lucid) Firefox/3.6.8 GTB7.1" My site is on a virtual private server, so it's entirely possible Apache is configured differently, but my site is returning a 304 "Not modified" and the referrer shown is the page that I'm launching the request from. Her log is showing a 200 "OK" but showing that there's no referrer. And it comes up as a 404 on the simulator site in any case. I'm completely at a loss here. -- Drew
Post #331,585 by folkert 8/20/10 5:51:44 AM Reply	I'd have to have first hand look at things just to get an idea of what is going on. And no... this ain't an offer, yet.
Post #331,600 by drook 8/20/10 12:53:16 PM Reply	I need to contact the host's support people In the shared server, I don't have access to the configuration pieces. The one that still confuses me is the DHFTIO site. From the emulator, it's showing a Blogger error page. I still can't figure out where something is cached that's pointing the domain at Blogger. -- Drew

Welcome to IWETHEY!