How to create a Web Scraper Feed

Web Scraper extracts posts from new links added to a site.

A typical case could be a competitor's press release page, or similar, with limited links where new items are added regularly.

Web Scraper does not look for previous page updates; it only parses new links. For updates on a page, try the VisualPing connection.

Web Scraper can not replace media monitoring.

Web Scraper is provided as-is and covers many types of pages. It will, however, not cover every case. If your desired website is not supported or not suitable, this will often be reported in the Status pane.

To create a Web Scraper feed:

1. Login to Wide Narrow and go to Inboxes.
2. On your DIY Inbox card: Click the More icon (three dots) and select InfoLab.
3. Click New Feed and select New Web Scraper Feed.
4. Enter a name to be used in Wide Narrow for the connection. A good practice is to use a name corresponding to the desired monitoring (e.g., Competitor X Press Releases).
5. Paste the desired URL in the URL field (e.g., https://competitorx.com/pressrelease).
6. Select the target Inbox for the RSS feed. Only applicable if you have more than one DIY Inbox.
7. Press Save.

Upon saving, you will be brought to the Status tab. The scraping is typically started immediately and will display the progress after a few seconds. If your page is not suitable, it will often be communicated on the feedback row. You may Close the status page at any time.

With the first fetch, it is not unusual that you get some undesired boilerplate articles, like About pages and menu links. These are often static and will not show up for subsequent fetches.

If your page is suitable, new posts will occur in Wide Narrow when new links occur on the page. The fetch interval is 24 h.

Wide narrow does not guarantee that the Web Scraper can scrape every kind of page.
For custom needs, feel free to reach out to our Information Specialists.

User Access: InfoLab and DIY Inboxes are available for all Superusers with access to Inboxes.