Diverse content extraction and scraping techniques are routinely used on web document collections by companies and research institutions alike. Being able to better qualify the contents allows for insights based on metadata (e.g. content type, authors or categories), better bandwidth control (e.g. by knowing when webpages have been updated), or optimization of indexing (e.g. language-based heuristics, LRU cache, etc.).
In short, metadata extraction is useful for different kinds of purposes ranging from knowledge extraction and business intelligence to classification and refined visualizations. It is often necessary to fully parse the document or apply robust scraping patterns, there are for example webpages for which neither the URL nor the server response provide a reliable way to date the document, that is find when it was written.
I regularly work on improving the extraction methods for the web collections at my home institutions. They are unique as they combine both the quantity resulting from broad web crawling and the quality obtained by carefully extracting text and metadata as well as rejecting documents that do not match certain criteria. In that sense, I already published work on methods to derive metadata from web documents in order to build text corpora for (computational) linguistic analysis, see for example Efficient construction of metadata-enhanced web corpora (2016).
The date however is a critical component, since it is one of the few metadata that are relevant both from a philological standpoint and in the context of information extraction for use in digital databases. It is crucial for my fellow lexicographers at the language center of the Berlin-Brandenburg Academy of Sciences to be able to determine with precision when a given word has been used for the first time and how its use evolves through time.
htmldate provides a simple and convenient way to extract the creation or modification date of web pages, based on HTML parsing and scraping functions:
- Starting from the header of the page, it uses common patterns to identify date fields.
- If this is not successful, it scans the whole document looking for structural markers.
- If no date cue could be found, it finally runs a series of heuristics on the content (text and markup).
The module is compatible with most of Python 2 and 3 versions. It takes the HTML document as input (string format) and returns a date if a valid cue could be found in the document. The output string currently defaults to ISO 8601 YMD format.
Install from package repository:
pip3 install htmldate
Direct installation of the latest version over pip is possible (but not thoroughly tested):
pip3 install git+https://github.com/adbar/htmldate.git
Unit tests with corresponding web page samples are available in the
A basic command-line interface is included:
$ wget -qO- "http://blog.python.org/2016/12/python-360-is-now-available.html" | htmldate 2016-12-23
For more information, type
In case the web page features clear metadata, the extraction is straightforward:
>>> import requests >>> import htmldate >>> r = requests.get('https://www.theguardian.com/politics/2016/feb/17/merkel-eu-uk-germany-national-interest-cameron-justified') >>> htmldate.find_date(r.text) '2016-02-17'
A more advanced analysis is sometimes needed:
>>> r = requests.get('http://blog.python.org/2016/12/python-360-is-now-available.html') >>> core.find_date(r.text) '# DEBUG analyzing: <h2 class="date-header"><span>Friday, December 23, 2016</span></h2>' '# DEBUG result: 2016-12-23' '2016-12-23'
In the worst case, the module resorts to a series of heuristics which can be deactivated (in “safe mode”) and tries to output a date. There is a trade-off in terms of granularity, current tests indicate a margin of error of 2-3 months.
>>> r = requests.get('https://github.com/scrapinghub/dateparser') >>> htmldate.find_date(r.text) '2017-07-01'
For more information see the readme page on GitHub.
Feedback and pull requests are welcome!