feeds timing out - slowing entire CMS

[alankeys]

Hi

Ive got a problem with RSS feeds that are slow to load.

My RSS feed started to time out and slow my whole site so I had to temporarily disable RSS feeds. Once they are disabled the site runs fine. Since titles and story content continue to be fetched ok from database and server, we recon the server and database and CMS scripts are ok.


Because I needed to disable the RSS on main site. I set up two sites with RSS feeds that are on the same server.

http://www.flightstation.co.uk/index1.php
http://www.homesplaces.co.uk/index1.php

The one on homesplaces is similar to the full setup that failed on main site [same database and stories with paths changed for homesplaces]. Fails completely to load and wont pass validation - simply times out.

The one on flightstation is similar to the full setup that failed on ukcitymedia - except, I deleted entries down to about 40 so that the feed loads ok and passes validation. [same database and stories with paths changed for flightstation].


I believe it must be something else within XML and RSS technology and not CMS, mySQL or server.


Ok

Im really struggling to find experts that can help me. Im just hoping that you may have an RSS / Feeds guru that may have an idea what the problem could be - or some testing tools - or anything.


Im totally stumped !

[Dormilich]

how big is the feed that fails loading? I ask because loading an XML file requires a massive amount of memory.

[alankeys]

about 400 articles of about 300 words each on average.

You will see I did I study of load in following thread
http://network-13.com/thread/2306-1-Slow-mySQL-requests

[alankeys]

about 400 articles of about 300 words each on average.

You will see I did I study of load in following thread
http://network-13.com/thread/2306-1-Slow-mySQL-requests



This is the rss.php file






<?php
/***********************************************************************
N-13 News is a free news publishing system
Copyright (C)
***********************************************************************/

define('ABSPATH', dirname(__FILE__) . '/' );

require_once(ABSPATH . 'db.php');
require_once(ABSPATH . 'config.php');

$_GET['feed'] = (empty($_GET['feed'])) ? '' : $_GET['feed'];
$feed = $_GET['feed'];
$rssexists = DataAccess::fetch("SELECT name,location,title,description,encoding,language,amounttodisplay,rsscode,nocats,feedid FROM " . NEWS_FEEDS . " WHERE name = ?", $feed);
if(!$rssexists){
die("RSS Feed doesn't exist.");
}
$all = $rssexists['0'];
$rsslanguage = $all['language'];
$rssdescription = $all['description'];
$rssencoding = $all['encoding'];
$rsstitle = $all['title'];
$rssamount = slash2($all['amounttodisplay']);
$newslocation = $all['location'];
$rsscode = $all['rsscode'];
$feedid = $all['feedid'];
$nocats = $all['nocats'];
if($rssamount == "0"){
$rssamount = '999999';
}

if(!$newslocation){
die("RSS Feed not configured. Please login to the admin area goto <b>Options</b> > <b>RSS Feeds</b>");
}
header("Content-type: text/xml");
$feedlocation = "TAKEN OUT BY POSTER" . $_SERVER['TAKEN OUT BY POSTER_HOST'] . $_SERVER['PHP_SELF'] . "?feed=" . $_GET['feed'];
echo '<?xml version="1.0" encoding="'.$rssencoding.'" ?>';
?>
<rss version="2.0" xmlns:atom="TAKEN OUT BY POSTER">
<channel>
<atom:link href="<?php echo $feedlocation; ?>" rel="self" type="application/rss+xml" />
<title><?php echo $rsstitle; ?></title>
<link><?php echo $newslocation; ?></link>
<language><?php echo $rsslanguage; ?></language>
<description><?php echo $rssdescription; ?></description>
<!-- RSS 2.0 file generated by N-13 News -->
<generator>N-13 News</generator>
<?php
$prefix = DataAccess::fetch("SELECT furlprefix FROM " . NEWS_OPTIONS . " WHERE 1");
$prefix = $prefix['0']['furlprefix'];
//select all categories this rss feed has been assigned
//then select all news posts assigned to those same cats
//if nocats = 1 also select all posts not assigned to cats
$sql = "SELECT title,story,shortstory,author,origauthor,ip,timestamp,allowcomments,short,approved,viewcount,rating, archivedate,neverarchive,archived,id,
" . NEWS_USERS . ".user AS authorname,
" . NEWS_USERS . ".avatar AS authoravatar,
commentcount AS comments
FROM " . NEWS_ARTICLES . " INNER JOIN " . NEWS_USERS . " ON " . NEWS_ARTICLES . ".author = " . NEWS_USERS . ".uid WHERE id IN (";
if($nocats == "1"){
$sql .= "SELECT id AS storyid FROM " . NEWS_ARTICLES . " WHERE id NOT IN (SELECT storyid FROM " . NEWS_GROUPCATS . " WHERE type = 'news') UNION ";
}
$sql .= "SELECT storyid FROM " . NEWS_GROUPCATS . " WHERE type = 'news' AND catid IN (SELECT catid FROM " . NEWS_GROUPCATS . " WHERE type = 'rss' AND storyid = ?) ) AND archived = '0' ORDER BY timestamp DESC LIMIT 0, $rssamount";
$newsstories = DataAccess::fetch($sql, $feedid);
foreach($newsstories AS $row){

if(FRIENDLY){
$fullurl = $newslocation . $prefix . $row['id'] . "/0/" . makefriendly($row[title]);
}else{
$fullurl = $newslocation . '?id=' . $row['id'];
}

$sql3 = "SELECT " . NEWS_CATS . ".name AS catname FROM " . NEWS_GROUPCATS . " INNER JOIN " . NEWS_CATS . " ON " . NEWS_GROUPCATS . ".catid = " . NEWS_CATS . ".id WHERE storyid = '?' AND type = 'news' ORDER BY " . NEWS_CATS . ".name ASC";
$cats = DataAccess::fetch($sql, $row['id']);
$categories = '';
foreach($cats AS $row3){
$row3['catname'] = (empty($row3['catname'])) ? '' : $row3['catname'];
$categories .= $row3['catname'] . CATDELIMITER;
}
$categories = substr($categories,0,strlen($categories) - strlen(CATDELIMITER));

$rsstemplate = $rsscode;
$rsstemplate = str_replace("{categories}",$categories,$rsstemplate);
$rsstemplate = str_replace("{title}",displayhtml($row['title']),$rsstemplate);
$rsstemplate = str_replace("{friendlytitle}", makefriendly($row['title']), $rsstemplate);
$rsstemplate = str_replace("{url}",$fullurl,$rsstemplate);
$rsstemplate = str_replace("{shortstory}",bbcode($row['shortstory'],0),$rsstemplate);
$rsstemplate = str_replace("{story}",bbcode($row['story'],0),$rsstemplate);
$rsstemplate = str_replace("{date}",date("D, j M Y h:i:s",$row['timestamp']),$rsstemplate);
$rsstemplate = str_replace("{timezone}",TIMEZONE,$rsstemplate);
$rsstemplate = str_replace("{timestamp}",$row['timestamp'],$rsstemplate);
$rsstemplate = str_replace("{author}",$row['authorname'],$rsstemplate);
$rsstemplate = str_replace("{avatar}",sprintf("<img src=\"%s\" />", $row['authoravatar']),$rsstemplate);
$rsstemplate = str_replace("{id}",$row['id'],$rsstemplate);
$rsstemplate = str_replace("{comments}",$row['comments'],$rsstemplate);
echo $rsstemplate;
unset($categories);
}
?>
</channel>
</rss>

[alankeys]

also here is config.php
and below that mysql.php






<?php
/***********************************************************************
N-13 News is a free news publishing system
Copyright (C) 2010 Chris Watt

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.If not, see <http://www.gnu.org/licenses/>.
***********************************************************************/

// Adds the security key image to the admin login form, 1 = on 0 = off
$image_verification = 0;

// Specify here which language should be used on the login module
$default_login_language = 'english';

// Specify here which language should be used on the index.php file
$default_index_language = 'english';

// Default options for the add news form
$newsform_options['toggle_archive'] = 0; // 0 = Hide archive options, 1 = Show the archive options
$newsform_options['never_archive'] = 1; // 0 = Never archive is not selected, 1 = Never archive is selected
$newsform_options['toggle_date'] = 0; // 0 = Hide date options, 1 = Date options are shown
$newsform_options['short_story'] = 0; // 0 = Hide short story, 1 = Short story section is shown by default
$newsform_options['allow_comments'] = 1; // Default comment options. 0 = No, 1 = Yes, 2 = Needs approval

// Enable or disable automatic creation of thumbnails when viewing uploaded images
// If you're having trouble viewing the thumbnails change this to 0
$imageupload_thumbnails = 1;

// Amount of characters to cut off the category delimiter when displaying categories
// Example 'Cat1, Cat2, Cat3, ' Notice the ending ', ' characters, specifying $catcutoff = 2 will get rid of that
$catcutoff = 2;

// Set the language used for formatting dates
setlocale(LC_ALL, 'eng');


/*------ NO NEED TO EDIT BELOW ---------*/
if(!defined('ABSPATH')){
define('ABSPATH', dirname(__FILE__) . '/' );
}

// if using the old db.php define constants here
if(isset($cfg)){
if(is_array($cfg)){
define('HOSTNAME', $cfg['hostname']);
define('USER', $cfg['user']);
define('PASS', $cfg['pass']);
define('DATABASE', $cfg['database']);
define('PORT', $cfg['port']);
define('SOCKET', $cfg['socket']);
if(empty($cfg['extension'])){
define('EXTENSION', 'mysql');
}
}
}


require_once(ABSPATH . '/functions.php');
require_once(ABSPATH . '/bbparser.php');
require_once(ABSPATH . '/recaptchalib.php');

$alloptions = DataAccess::fetch("SELECT * FROM " . NEWS_OPTIONS . " WHERE 1");
$registrationverification = $alloptions['0']['registrationverification'];

define('SPAMTIME', $alloptions['0']['spamtime']);
define('SPAMMSG', $alloptions['0']['spammsg']);
define('COMMENTMSG', $alloptions['0']['commentmsg']);
define('REGCOMMENT', $alloptions['0']['regcomment']);
define('COMMENTAPPROVEMSG', $alloptions['0']['commentapprovemsg']);
define('TIMEZONE', $alloptions['0']['timezone']);
define('BANNEDMSG', $alloptions['0']['bannedmsg']);
define('COMMENTSORDER', $alloptions['0']['commentsorder']);
define('CATDELIMITER', $alloptions['0']['catdelimiter']);
define('COMMENTSNOTIFY', $alloptions['0']['commentsnotify']);
define('REGISTERNOTIFY', $alloptions['0']['registernotify']);
define('EMAILNOTIFY', $alloptions['0']['emailnotify']);
define('UNAPPROVEDNOTIFY', $alloptions['0']['unapprovednotify']);
define('USEIMGVERIFICATION', $alloptions['0']['useimgverification']);
define('UPLOADPATH', $alloptions['0']['uploadpath']);
define('SALT', $alloptions['0']['salt']);
define('COMMENTSLENGTH', $alloptions['0']['commentslength']);
define('REGISTRATIONALLOW', $alloptions['0']['registrationallow']);
define('REGISTRATIONACCESS', $alloptions['0']['registrationaccess']);
define('REGISTRATIONVERIFICATION', $alloptions['0']['registrationverification']);
define('PUBLICKEY', $alloptions['0']['publickey']);
define('PRIVATEKEY', $alloptions['0']['privatekey']);
define('TAIL', $alloptions['0']['tail']);
define('ENABLETAIL', $alloptions['0']['enabletail']);
define('FRIENDLY', $alloptions['0']['furlenabled']);
define('FILETIME', $alloptions['0']['filetime']);
define('DELETEOWNCOMMENTS', $alloptions['0']['deleteowncomments']);
define('LOGINVERIFICATION', $alloptions['0']['loginverification']);
define('NOACCESSMSG', $alloptions['0']['noaccessmsg']);
define('NEWSTIME', $alloptions['0']['newstime']);
define('COMMENTSTIME', $alloptions['0']['commentstime']);
define('TEMPLATE', $alloptions['0']['template']);
define('NPPAGE', $alloptions['0']['nppage']);
define('CPPAGE', $alloptions['0']['cppage']);
define('NEWSORDER', $alloptions['0']['newsorder']);
define('SUFFIX', $alloptions['0']['furlextension']);
define('NEWSSUFFIX', $alloptions['0']['furldirname']);
define('PREFIX', $alloptions['0']['furlprefix']);
define('IMAGEPATH', $alloptions['0']['scriptpath'] . "images/");
define('ADMINPATH', $alloptions['0']['scriptpath'] . "admin.php");
define('SCRIPTPATH', $alloptions['0']['scriptpath']);
$version = $alloptions['0']['version'];
$url = explode("/",$_SERVER['REQUEST_URI']);
$imageuploaddir = 'uploads/'; #slash at the end is important!
unset($alloptions);

define('SMILIES', serialize(DataAccess::fetch("SELECT keycode, path FROM " . NEWS_SMILIES)));
define('FILTERS', serialize(DataAccess::fetch("SELECT filter, alt FROM " . NEWS_FILTER)));


#archive any posts that need to be.
DataAccess::put("UPDATE " . NEWS_ARTICLES . " SET archived = ? WHERE archivedate <= ? AND neverarchive = ?", "1", time(), "0");
?>




end of config.php
start of mysql.php










<?php
/***********************************************************************
N-13 News is a free news publishing system
Copyright (C) 2010 Chris Watt

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.If not, see <http://www.gnu.org/licenses/>.
***********************************************************************/

class DataAccess {
function establish_db_conn(){
$connection = mysql_connect(HOSTNAME.":".PORT,USER,PASS);
$db = mysql_select_db(DATABASE);
return $connection;
}

function put() {
$args = func_get_args();
$sql = array_shift($args);
$link = DataAccess::establish_db_conn();

if(count($args) > 0){
$sqlparts = explode("?", $sql);
$newsql = '';
$x = 0;
foreach($sqlparts AS $part){
if(isset($args[$x])){
$argpart = $args[$x];
}else{
$argpart = '';
}
$argpart = mysql_escape_string($argpart);
$newsql .= $part;
if($x < count($args)){
$newsql .= sprintf("'%s'", $argpart);
}
$x++;
}
}else{
$newsql = $sql;
}
$query = mysql_query($newsql);
if($query){
$_SESSION['insert_id'] = mysql_insert_id();
return true;
}else{
$error = mysql_error($link);
die('<strong style="color: #AA0000">Error</strong>: ' . $error . ' <br /><strong style="color: #AA0000">Statement</strong>: ' . $sql . ' <br /><strong style="color: #AA0000">Arguments</strong>: ' . implode(",", $args));
return false;
}

}
function fetch() {
$args = func_get_args();
$sql = array_shift($args);
$link = DataAccess::establish_db_conn();
if(count($args) > 0){
$sqlparts = explode("?", $sql);
$newsql = '';
$x = 0;
foreach($sqlparts AS $part){
if(isset($args[$x])){
$argpart = $args[$x];
}else{
$argpart = '';
}
$argpart = mysql_escape_string($argpart);
$newsql .= $part;
if($x < count($args)){
$newsql .= sprintf("'%s'", $argpart);
}
$x++;
}
}else{
$newsql = $sql;
}

$query = mysql_query($newsql);

if($query){
$results = array();
$x = 0;
while($row = mysql_fetch_assoc($query)){
foreach($row AS $key => $val){ $results[$x][$key] = $val; }
$x++;
}
return $results;
}else{
$error = mysql_error($link);
die('<strong style="color: #AA0000">Error</strong>: ' . $error . ' <br /><strong style="color: #AA0000">Statement</strong>: ' . $sql . ' <br /><strong style="color: #AA0000">Arguments</strong>: ' . implode(",", $args));
return false;
}
}
}
?>

[alankeys]

Dormilich


Have you any ideas about this.

I would be obliged if you could indicated any issues about the coding ....

[Dormilich]

currently I have no idea besides the XML eating your memory.

[alankeys]

hmm...
Which code do your want ?
Ive posted
rss.php
config.php
mysql.php ?

[alankeys]

Dormilich

You asked me to post the code. I did that.
Were there any issues you spotted in the code ?

Is it a different file for XML your wanted me to post ??

[alankeys]

We have noticed that, if you run the feed in webbug V5 that you get strange (hex codes?) characters appear.....

So far, cant see anything in the code that could explain this. Also, a search through the database (opened in wordpad) doesnt directly find the characters. They could aguably be within session ID's but I think that would be a red herring.

any further comments and ideas appreciated.


27
<?xml version="1.0" encoding="UTF-8" ?>
5a
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
<channel>
<atom:link href="
3c
http://www.homesplaces.co.uk/news/rss.php?feed=property_news
35
" rel="self" type="application/rss+xml" />
<title>
f
ttttttttttttttt
12
</title>
<link>
2f
http://www.homesplaces.co.uk/news/fullstory.php
15
</link>
<language>
5
en-us
1c
</language>
<description>
10
dddddddddddddddd
63
</description>
<!-- RSS 2.0 file generated by N-13 News -->
<generator>N-13 News</generator>
--------------------------------------------------------------------------------


alankeys
View Public Profile
View Extended RPG Stats
Challenge This User To Battle
Send a private message to alankeys
Find all posts by alankeys
Find all threads by alankeys
Add alankeys to Your Buddy List