Ufdjrw

I am building a import tool just like medium.com story import tool so far i have used this code

include('includes/import/simple_html_dom.php');
// get DOM from URL or file
$html = file_get_html('https://neilpatel.com/blog/starting-over/');

// find all link
foreach($html->find('a') as $e) 
 echo $e->href . '<br>';

// find all image
foreach($html->find('img') as $e)
 echo $e->src . '<br>';

// find all image with full tag
foreach($html->find('img') as $e)
 echo $e->outertext . '<br>';

// find all div tags with id=gbar
foreach($html->find('div#gbar') as $e)
 echo $e->innertext . '<br>';

// find all span tags with class=gb1
foreach($html->find('span.gb1') as $e)
 echo $e->outertext . '<br>';

// find all td tags with attribite align=center
foreach($html->find('td[align=center]') as $e)
 echo $e->innertext . '<br>';

// extract text from table
echo $html->find('td[align="center"]', 1)->plaintext.'<br><hr>';

// extract text from HTML
echo $html->plaintext;

But this scrape the whole page is it possible to just find and scrape only the main content like the medium import tool doing for any link

Kindly solve this problem and how can i achieve this kind of result

I'm not completely sure what you are asking / trying to do.. But I'll give it a try.

You are trying to Identify the main content area - To scrape only the needed information without any garbage or unneeded content.

My approach is to use the common structures and good practices of well formatted HTML pages. Consider this:

• The main article will be encapsulated in a unique ARTICLE tag on the page.


• The H1 tag on the article will be its header.


• We know that there are some repeating ID's used such as (main_content, main_article, etc..).

Summarize those rules on your targets and build an Identifiers list sorted by priority -> Then you can try and parse the target until one of the identifiers will be found - which indicates that you identified the main content area.

Here is an Example -> using the URL you provided:

$search_logic = [
 "#main_content",
 "#main_article",
 "#main",
 "article",
];

// get DOM from URL or file
$html = file_get_contents('https://neilpatel.com/blog/starting-over/');
$dom = new DOMDocument ();
@$dom->loadHTML($html);

//
foreach ($search_logic as $logic) 

 $main_container = null;

 //Search by ID or By tag name:
 if ($logic[0] === "#") 
 //Serch by ID:
 $main_container = $dom->getElementById(ltrim($logic, '#'));
 else 
 //Serch by tag name:
 $main_container = $dom->getElementsByTagName($logic);
 

 //Do we have results:
 if (!empty($main_container)) 

 echo "> Found main part identified by: ".$logic."n";
 $article = isset($main_container->length) ? $main_container[0] : $main_container; // Normalize the container.

 //Parse the $main_container:
 echo " - Example get the title:n";
 echo "t".$article->getElementsByTagName("h1")[0]->textContent."nn";

 //You can stop the iteration:
 //break;

 else 
 echo "> Nothing on the page containing: ".$logic."nn";
 

As you can see the firs to ID's were not found so we keep trying down the list until we hit the result we want -> a good set of those tagnames / ID's will be good enough.

Here is the result:

> Nothing on the page containing: #main_content

> Nothing on the page containing: #main_article

> Found main part identified by: #main
 - Example get the title:
 If I Had to Start All Over Again, I Would…

> Found main part identified by: article
 - Example get the title:
 If I Had to Start All Over Again, I Would…

Hope I helped.