XPath (XML Path Language) is a W3C expression language for addressing and selecting parts of an XML document, and it can also navigate HTML. Its defining feature is the path expression, such as //a/@href, which walks the document tree to reach elements, attributes or text. Scraping tools such as Scrapy support XPath for pulling data out of downloaded pages.

The current version, XPath 3.1, became a W3C Recommendation on 21 March 2017. It works on XML and, in this version, JSON data.

XPath Syntax Basics

  • Path: //title selects every title element anywhere in the document.
  • Text: //title/text() returns the text inside those elements.
  • Attributes: @ selects an attribute, so //a/@href returns link targets.
  • Predicates: Square brackets filter results, so //div[@class="price"] keeps only div elements with that class.

XPath in Web Scraping

Scrapy selectors accept both XPath and CSS, and Scrapy translates CSS queries into XPath internally using the cssselect library. Its selectors use the lxml library under the hood. In browsers, XPath can navigate the DOM of HTML and SVG documents through the XPathExpression interface, as an alternative to methods like querySelectorAll().

XPath Functions and XSLT

XPath is designed to be embedded in a host language such as XSLT. The W3C defines one function library, in the fn namespace, for XPath, XQuery and XSLT. Functions useful in scraping include:

  • contains() and starts-with(): Substring matching.
  • normalize-space() and string-length(): Work on string values.
  • count(): Counts items.

For example, //a[contains(@href, "product")] keeps only links whose URL contains "product".