As an SEO for a small brand marketing agency, the unfortunate conundrum which resulted in the collapse of ilovepage1, has left some scrambling for a decent alternative.
You might recognize Ilovepage1 as the popular and useful tool that helped SEO’s scrape Google Search Results pages for “Allintitle:” results data (i.e. “about 1,000 search results”). For those that do not, unfortunately its dust in the wind 🙁 Since then, I have spent many tireless hours working on a solution. After all, the more useful data we can find to give us that competitive edge, the better, right?
It took some digging. There are not many post about this topic and I repeatedly found myself lost deep in the jungles of GitHub and StackExchange looking for clues that could solve this very seemingly simple problem. I was ALMOST successful at creating the simplest, easy to customize solution using the XML Library in Excel VBA. I WAS SO CLOSE!
In that, I mean I had successfully scraped the data I needed, deposited it cleanly into an excel spreadsheet, found a easy way to extract it and process it in a usable format as actionable keyword data, all within the same step. This completely cut out the middle man and forwent the need for any program other than a Keyword Research tool and the Excel spreadsheet itself.
I don’t know about you, but exporting and importing back and forth became a real pain for me. This “solution” looked to be a gold mine of automation and time saving. It worked the first time flawlessly, yielding data for about 300 results for the first run. But to my despair, it progressively got worse.
As with anything that seems to good to be true, there was one important factor that prevented this from being universally usable, Google hates bots. Which was effectively what this program was.
I did everything I could to spoof the bot into appearing as human as possible in an attempt to convince Google to avoid blocking out my bot with the Captcha. I looked into VPNs, time interval manipulation, Captcha solvers, you name it. Google was just too far ahead. My latest attempt at running the program yielded only 8 results before Google recognized it and blocked it out with a Captcha. I buried project.
It wasn’t until a few months had passed and I realized that Ilovepage1 was discontinued and not even a viable backup anymore. Now with time not on my side, I was forced to hastily find another option. I needed it for my clients after all. Recently, I came across a solution that presents the use of Google Sheets importxml lookups (which is very similar to what I had been using in Excel) to scrape the very same data. Here is a link to the original page in which I found it –> Bulk Check Google Results Count with Google Docs
Why did this work? Hard to say, other than the fact that Google owns both ends of the line when comes to the data scrape as opposed to my version where Excel seemingly presents some invasive species to the search engine. All that I care about now, is that this appears to work beautifully and I was even able to craft my own version of the tool for the specific purposes that I use them for. If you would like a copy of my version to try it out, just click on the image of the box to the right.
I enjoy a challenge, I would consider helping you design a perfect version of this tool to fit your SEO needs, just leave a comment below.