Introduction

• The syntax of Scrapeium is meant to be simple, free and “speakable”

• The syntax will be largely inspired by JSON, GraphQL

• Like GraphQL, the structure of the query greatly influences the shape/structure of the data that is returned and how the data is scraped from the website

• This structure is described using entities called “blocks”, and the concept of blocks are key in Scrapeium

• All queries start with a single block

• Comments are defined using the “#” character

  • only single line comments supported at the moment

  # this is a comment
  

Blocks

• A block is a structure / entity (would I call it a structure?) used to define the shape/structure of the returned queried/scraped data
• Blocks are an essential part of Scrapeium and are what makes Scrapeium as dynamic/flexible as it is.
• Some blocks can be nested in each other (see below)
• All block have their own scope and they also inherit the scope of any parent blocks
  • Can be seen as values cascading down and changing with scope
• There are 3 types of blocks in Scrapeium:

  • Array blocks

    • Array blocks represent an array of values

    • Array blocks can be used to query many elements on a website

    • an Array block can either have an expression block or object block as a child block

      • Use expression block if you want an array of primitives (string, numbers)
      • Use a object if you want an array of objects (see Object Blocks section for more details)

    • Every array block requires a block prefix

      • see below

      // this is an example of a block prefix
      "h1" > [...] 
      // in this example, this tells the assocciated array block to query over every h1 tag and "loop" over each one with the provided child block
      // this means in the result array, there will be an item for every h1 on the page (the value and type of the item being determined by the kind of block used)
      

    • Using the selector provided from the defined block prefix, the array block will query all elements that match that selector and for each element, the given child block will be executed under the context of that element (meaning the :element variable will be set to that variable and variables like :inner_text will be defined in the scope of the child block, refering to that element)

    • The best way to think of an array block is a for each loop, where the block prefix is the condition for the loop (or array of (potential) elements) and the child block is the ”function” executed for each of the elements (and child block receiving the context defined by the element)

    • Example

    
    // this query queries all ".item" elements and loops over them with an expression block
    // this query results in an array of strings (the inner text of every ".item" element)
    
    // <h1 class="item">hello</h1>
    // <h1 class="item">hi</h1>
    
    ".item" > [(
    	// all the variables in this block refer to an abitrary element that matches the ".item" selector
    		read :inner-text
    )]
    // ["hello", "hi"] 
    
    // this query results in an array of objects, where the each object has a name key with a value of the inner text of each queried element
    ".item" > [{
    	greeting = read :inner-text,
    }]
    // [{greeting: "hello"}, {greeting: "hi"}]
    

  • Object blocks

    • an object block is used to structure pairs of keys and values
    • the syntax for key values is key = value
    • results in an object with the matching structure
    • keys are normal identifiers while values can be any block type
    • Like all blocks, object blocks define their own scope and inherit the scope of their parents. If the object block has a block prefix, it will also inherit the context defined by it
    • if the parent to an object block is an array block, the object block acts as a “model” for multiple items in a potential array (this is the most likely use of object blocks)
    • Examples

    
    // <div id="name">Ben</div>
    // <div id="age">10</div>
    // <div class="jobs">Developer</div>
    // <div class="jobs">Designer</div>
    // <div class="jobs">Manager</div>
    {
    	person = {
    		name = (
    			query "#name"
    			read :inner-text
    		)
    		age = (
    			query "#age"
    			read :inner-text
    		)
    		jobs = ".jobs" > [read :inner_text]
    	}
    }
    
     
    // { person: { name: "ben", age: "10", jobs: ["Developer", "Designer", "Manager"] } }
    

  • Expression blocks

    • Expression blocks are used to execute statements that interact with the website and return a primitive value (see below for detailed list of statements)
    • Expression block are designed to work on an single element at a time and the return a value
    • Like all blocks, expression blocks define their own scope and inherit the scope of their parents. If the expression block has a block prefix, it will also inherit the context defined by it
    • returns the result of the last statement
    • has to return a primitive value (explained below)
    • Expression blocks MUST return a value that is not null or else an error is thrown
    • if you only had one expression to evaluate, you could use a short hand and omit the brackets
    • can be used in conjunction with other blocks
    • Example

    // <div id="message">This is cool</div>
    
    // simple expression block within an object block
    {
    	message = (
    		query "#message"
    		read :inner_text
    	)
    }
    
    // the same expression block but using a block prefix to make it shorter
    {
    	message = "#message" > (
    		read :inner-text
    	) // like a shorthand syntax almost
    }
    
    // even shorter way to do it
    {
    	message = "#message" > read :inner-text
    }
    
    // can also have expression block in array blocks to produce an array of primitive values
     
    /*
    <div class="messages">This is cool</div>
    <div class="messages">Hi</div>
    <div class="messages">hello</div>
    */
    ".messages" > [read :inner_text]
    // ["This is cool", "Hi", "hello"]
    
    // same as 
    ".messages" > [(
    	read :inner_text
    )]
    
    // message block by itself
    // syntax like this is generally not recomended
    // using an object block is best practice as the value is named with a key
    (
    	query "#message"
    	read :inner-text
    )
    
    // result = "This is cool"
    

Block prefix

• A block prefix is an entity that is used to define the context of which any block will operate under
• For expression blocks and object blocks they define the :element variable that the block works with (unless explicily changed) and all its accosiated variables (like :inner_text and :id)