Event Scraper

Asynchronous scraping as fast as possible!

Installing:

npm install event_scraper

Starting example:

import { Event, EventListener, EventHandler, IContext } from 'event_scraper';

class AsyncScrapeEvent extends Event {
  context: IContext = {
    callerEvent: this,
    proxied: false,
    numberOfTry: 1
  }

  public async event(proxy?: string) {
    // do something asynchronous...
    return result;
  }
}

const event = new AsyncScrapeEvent();

class MyEventListener extends EventListener {
  eventToListenTo = AsyncScrapeEvent.name;

  onSuccess(context: IContext, result: any): void {
    // do something with result
    
    // start new events as:
    EventHandler.scheduleEvent(newEvent);
  }
}
const eventListener = new MyEventListener();

// set up everything:
const eventHandler = new EventHandler(5);


const proxies = []; /** if you have proxies... */
eventHandler.setEventListener(eventListener);
eventHandler.setProxies(proxies);
eventHandler.startEvent(event);

Description

This module operates with a basic EventEmitter under the hood. The goal of the project is to create events, all of which can do anything asynchronous while the relationship between them are leveraged by the event-listeners we set. The addition which makes it more than just a regular event-emitter, is that you can feed in a given set of proxies, and every time an event requires a proxy, it will automatically assign one to it completely randomly. There are always as many proxied asynchronous events as many proxies you feed in.

The other extra is that you can set how many retries it should do. This way if any event fails, it'll automatically reschedule it.

Documentation

Event class

Every time you create a new event, you should extend the Event class. You'll be required to create a context parameter. There are three parameters required here. These are:

You should also define the asynchronous event function, which can get an optional parameter, the proxy as a string if you set the proxied context parameter as true.

If you get a proxy it'll be a string in the following format:

http://proxy.ipAddress:proxy.port // http:/141.12.16.59:4400

You can put anything in the context parameter as you'd like, these can be used later at the event listeners, since the functions you can override there will all get you extended context. Therefore if you'd like to transfer data from the event to the appropriate event listener, you should put it into the context.

EventListener class

Every event listener you'd like to set up has to extend from the EventListener class. Here you have six methods to override. These are:

When setting up the EventListener class, you have to define a string property called eventToListenTo. This value of this should be the name property of the Event class to listen to. See example above.

If the logger functions don't get overridden, nothing will be logged.

EventHandler class

This is the interface through which you can set up the listeners, events and proxies as well. Also it gives you a handy function to schedule new events into the pool as well. When instantiated the parameter that it needs will be the total number of times it will try to do each event in case of failures.

And there is an abstract method defined as well:

IContext

This is the interface of an event's context. If you'd like to add anything to the context, you should extend this interface and after that everywhere where it requires the type IContext, you should just use your extended context.

TODO: Get rid of the numberOfTry... Testing, testing, testing...

License

MIT