MySQL speed improvements

MyISAM seems to be faster than InnoDB with the queries that I've been using for my web spider. I switched over my tables and found that my crawls were running 3 times faster because MyISAM supports INSERT DELAYED.

The Queries
For the example, we will using entry #383558 (http://en.wikipedia.org/wiki/Category:World_Wrestling_Entertainment_alumni).

SELECT uID, uURL FROM urls WHERE uError = 0 AND uUpdated < DATE_SUB(NOW() , interval 12 hour ) ORDER BY rand() LIMIT 1

Select a random URL from the url table that hasn't errored and hasn't been updated in over 12 hours.

UPDATE urls SET uUpdated = NOW() WHERE uID = 383558 LIMIT 1

Set the last updated time to now() in the url table.

INSERT DELAYED INTO data (dURLID, dData) VALUES ( 383558, 'Category:World Wrestling Entertainment alumni - Wikipedia, the free encyclopedia\n /**/...' ) ON DUPLICATE KEY UPDATE dData=VALUES(dData)

This inserts (or updates) the current page's text into the data table.

INSERT DELAYED IGNORE INTO urls (uURL, uAdded, uSiteID) VALUES ( 'http://en.wikipedia.org/favicon.ico', NOW(), 383558 ),( 'http://en.wikipedia.org/wiki/Kurt_Angle', NOW(), 383558 ),( 'http://en.wikipedia.org/wiki/Bryan_Clark', NOW(), 383558 ),( 'http://en.wikipedia.org/wiki/Peter_Gasperino', NOW(), 383558 ),( 'http://en.wikipedia.org/wiki/Robert_Horne_%28wrestler%29', NOW(), 383558 )

This inserts more urls to the url table and records the parent url.

UPDATE urls SET uError = 1 WHERE uID = 383558 LIMIT 1

This marks the page as an error (e.g. 404). It will not be spidered in the future without this mark being removed.


This is an extract from a typical set of querys that would get executed after a crawl. Obviously the dData wouldn't be truncated and there would be a lot more urls lol. tacos. l8r

Using "IF Comments" in PHP

Here's a nifty hack using comments to quickly enable/disable a block of code. I haven't seen examples of this before so I feel special. Here we go.

Teh Test
Suppose you have the following code:

$data = get_crap();

echo "Debug:"
print_r($data);

I store some data to $data and as a debug I want to view the contents of $data. I want to disabled the debug quickly. This is going to look weird at first but I will explain.

Example Code:

$data = get_crap();

///*
echo "Debug:"
print_r($data);
//*/

///* is a valid comment. it starts with //. php ignores the /*
The echo and print_r work because only the line above was commented.
//*/ is a valid comment. it starts with //. php ignores the */

To disable execution of code:
(Remove the very first set of //'s)

$data = get_crap();

/*
echo "Debug:"
print_r($data);
//*/

/* is a multi-line block comment. php ignores everything until it finds a matching */
The echo and print_r are not seen by the compiler.
//*/ was ignored. Execution resumes after the */

Wait a second...
Why not just do this: ?

$data = get_crap();

/*
echo "Debug:"
print_r($data);
*/

Why not just use /* */ why /* //*/ ??

Beacuse! if you remove the first /* to 'uncomment' without removing the */ (an extra step by the way) you get this:

$data = get_crap();


echo "Debug:"
print_r($data);
*/

And this:

Parse error: parse error in c:\www\project\file.php on line n

meh, l8r