Architecture

A question mark (?) after a word, phrase, or diagram component means that item is optional.

OpenScraping Context#

The below diagram illustrates a common example of how OpenScraping is used in a broader context/on a macro scale.

Scraper
Create Job
Distributes Job to an Agent
Run, and store in?
Retrieve Data
Check status
Notify Job completed, with data?
Agent
OpenScraping Job API
Data Store
Client

Agent Execution#

This diagram shows a visual representation of the rough execution flow of any given OSP agent.

Agent
Matcher Processing
Fetch job for agent
Store relevant data
Requestor
Matcher 2
No Match, end
... Matcher N
Parser?
Matcher 1
Extractor
Generator?
Job Service
Data Service

The Requestor accesses the network. TODO: Matchers are run in parallel to improve performance. Each Matcher Processing operation is run sequentially and synchronously on that Matcher's thread.

Edit this page