Scraping with Groovy (II): Geb

The definition of Geb from its web site is "very groovy browser automation… web testing, screen scraping and more"

I've been using Geb mostly for functional testing (or more accurately for acceptation testing) but it could be also a very powerful weapon for scraping web pages. We saw how XmlSlurper is enough for scraping simple pages when no interaction is needed. But things normally get complicated.

Geb can use your favorite browser engine underneath to scrap really complicated pages with javascript, css... etc. But you if you like you can also use HtmlUnit like your browser engine. This way you can execute your script in any environment (No need to install any browser in every environment). Said so, it's also true that HtmlUnit has some limitations, specially if the page you are scraping used javascript in a very intensive way.

Geb uses a really simple DSL and a "JQuery like" syntax (with the $ character) for traversing html pages. This could be very handy sometimes when trying to look for a given DOM node. If you know how to get it with JQuery the solution with Gen will be pretty the same (read the documentation anyway).

As this is a really short example there's no need of higher abstraction, just getting a page, traversing the page, trying to get the relevant elements. But when trying to do the same stuff through several pages it's gonna be a mess. Geb solves those situations with the "Page Object Pattern".

The following code goes to the Grails plugin portal and looks for plugins related to the "Geb" term. For this example I've used latest Geb release (0.7.0). It's just a groovyConsole script, you can copy&paste and then run it (Dependencies are declared with Grapes).

@Grapes([  
    @Grab("org.codehaus.geb:geb-core:0.7.0"),
    @Grab("org.seleniumhq.selenium:selenium-htmlunit-driver:2.23.1"),
    @Grab("org.seleniumhq.selenium:selenium-support:2.23.1")
])
import geb.Browser

Browser.drive{
    // driver.webClient.javaScriptEnabled = true
    go "http://grails.org/plugins/"      
                
    $("input", name: "q").value("geb")
    $("input", class: "searchButton").click()   
             
    waitFor(10){ $("div", class:'currentPlugin') }
        
    def pluginNames = $("div", class:'currentPlugin').collect{div->
        div.find("h4").find("a").text()    
    }
    
    println pluginNames
            
}
    

• Start the script inside the DSL and go to the page we want to start our scraping work. This time I've disabled the javascript interaction because form submission can work without javascript.

Browser.drive{  
 // driver.webClient.javaScriptEnabled = true
go "http://grails.org/plugins/" 
//...
}

• Look for relevant elements from the form. Then put some value inside the input text and click the form button.

$("input", name: "q").value("geb")
    $("input", class: "searchButton").click()   

• In this moment the "browser" is sending the request and getting the response. We want to be sure the response has been received before continuing asking. In this case I waited for 10 seconds the given div to show up (by default is 5 seconds).

waitFor(10){ $("div", class:'currentPlugin') }

• Once the page is ready we want to get the list of the retrieved plugins' names

def pluginNames = $("div", class:'currentPlugin').collect{div->
        div.find("h4").find("a").text()    
    }
    
    println pluginNames

Bottom line, I would use Geb :

• For testing
• For scraping really complicated pages in an elegant way thanks to its DSL

Resources:

• Geb web site
• Geb latest documentation
• HtmlUnit project

Scraping with Groovy (I): XmlSlurper

There're times when a man have to scrap web pages, and when those times come... you'd better have Groovy.

Here I'm gonna write about some things I've learned about scraping with Groovy. I will use core functionalities like XmlSlurper and GPath, and some more advanced tools like Geb.

Right now I'm following the UEFA EURO 2012 (Go Spain!!) and I want to get the final list of the teams playing the quarter finals. I already now the url so I just have to parse the page and take the list of the first table. In this post I'm gonna explain how to use XmlSlurper to do so.

Time to analysis:

First thing I have to do is to locate the html where the results are. Well, activate your firebug (Firefox) or   "Inspect Element" on Safari, and inspect the html. In this case we have the following structure:

• (1) All phases are group inside div elements with class "box boxSimple" so I need the first
• (2) Inside that div we have to look for a list (ul element)
• (3) Every element in the list is assigned to a team. I need the name of each team. They are located inside the attribute @title inside the first anchor available inside each element of the list.

Before coding:

We are gonna use XmlSluper to parse an HTML page, but there's one problem XmlSlurper only parses XML. Don't worry, to fix that you need the NekoHtml parser. It converts an HTML page into an XHTML page. Then you'll be able to parse the page and retrieve the information we are looking for.

Let's code:

I've used groovyConsole to do it. Pay attention to the dependency.

@Grapes([
    @Grab(group='net.sourceforge.nekohtml', module='nekohtml', version='1.9.14')
])
import org.cyberneko.html.parsers.SAXParser
import groovy.util.XmlSlurper

/* Getting the xhtml page thanks to Neko SAX parser */
def imdbPageXml = new XmlSlurper(new SAXParser()).
    parse("http://www.uefa.com/uefaeuro/season=2012/teams/index.html")    
    
def teams = 
 /* (1) Go to first <div class='box boxSimple'> */
    imdbPageXml.'**'.find{it.name() == 'DIV' && it.@class=='box boxSimple'}.
    /* (2) Folow the path */
        DIV[1].UL.LI.collect{li-> 
         /* (3) For each element in list get the title of the first anchor */
            li.'**'.find{it.name() == 'A'}*.@title
        }.flatten()

println teams

Once XmlSlurper returns a navigable instance of GPathResult the object can be traversed using  GPath:

• Go to first div with class 'box boxSimple'

imdbPageXml.'**'.find{it.name() == 'DIV' && it.@class=='box boxSimple'}.

To avoid to put explicit all the way to the element I'm looking for I started using '**' and then a query. That line means "Look everywhere in the document and take the first element with name div and having the attribute class with value 'box boxSimple'.  Remember '**' could be really useful.

I use find instead of findAll to get only the first element. Also notice all HTML elements are referred in upper case.

• Folow the path to the elements of the list

DIV[1].UL.LI.collect{li-> 

This path is very short and I didn't know how to do it shorter, so I put the path explicitly.

• For each element get the title of the first anchor

li.'**'.find{it.name() == 'A'}*.@title

"Everywhere inside this 'li' element look for an element with name 'a'. Then get its @title attribute value.

Bottom line, use XmlSlurper for scraping when:

• If you're scraping WebServices returning well formed XML
• If you're scraping simple pages and you're not going to interact with the page. Then use XmlSluper with the Neko parse to ensure the page conversion to XML.

But:

• If you're going to interact with the page and still want to do it with Groovy then I'd recommend you to use Geb

Resources:

1. Groovy Gpath page with examples
2. Groovy page on how to use XmlSlurper (GPath examples)
3. XmlSluper javadoc