Desynchronized Spider

This abstract class can be extended to produce new plugins that do not require a system-wide configuration synchronization.

pathspider.desync

class pathspider.desync.DesynchronizedSpider(worker_count, libtrace_uri, args, server_mode=False)[source]
configurator()[source]

Since there is no need for a configurator thread in a desynchronized spider, this thread is a no-op

connections = []
classmethod register_args(subparsers)[source]
worker(worker_number)[source]

This function provides the logic for configuration-synchronized worker threads.

Parameters:

worker_number (int) – The unique number of the worker.

The workers operate as continuous loops:

  • Fetch next job from the job queue

  • Perform pre-connection operations

  • Acquire a lock for “config_zero”

  • Perform the “config_zero” connection

  • Release “config_zero”

  • Acquire a lock for “config_one”

  • Perform the “config_one” connection

  • Release “config_one”

  • Perform post-connection operations for config_zero and pass the result to the merger

  • Perform post-connection operations for config_one and pass the result to the merger

  • Do it all again

If the job fetched is the SHUTDOWN_SENTINEL, then the worker will terminate as this indicates that all the jobs have now been processed.