superduperdb.backends.ibis.cdc package#

Submodules#

superduperdb.backends.ibis.cdc.base module#

class superduperdb.backends.ibis.cdc.base.IbisDBPacket(ids: List[str], query: Serializable | None = None, event_type: DBEvent = DBEvent.insert)[source]#

Bases: Packet

A base packet to represent message in task queue.

event_type: DBEvent = 'insert'#
ids: List[str]#
query: Serializable | None = None#

superduperdb.backends.ibis.cdc.listener module#

class superduperdb.backends.ibis.cdc.listener.IbisDatabaseListener(db: Datalayer, on: Table, stop_event: Event, identifier: str = '', timeout: float | None = None, strategy: PollingStrategy | LogBasedStrategy | None = None)[source]#

Bases: BaseDatabaseListener

It is a class which helps capture data from ibis database and handle it accordingly.

This class accepts options and db instance from user and starts a scheduler which could schedule a listening service to listen change stream.

This class builds a workflow graph on each change observed.

__init__.

Parameters:
  • db – It is a superduperdb instance.

  • on – It is used to define a Collection on which CDC would be performed.

  • stop_event – A threading event flag to notify for stoppage.

  • identifier – A identifier to represent the listener service.

  • strategy

    Used to select strategy used for listening changes options:

    PollingStrategy (This strategy polls table every

    frequency seconds, more info at superduperdb.cdc.cdc.PollingStrategy)

    LogBasedStrategy (Not implemented yet)

DEFAULT_ID: str = 'id'#
EXCLUSION_KEYS: Sequence[str] = ['id']#
IDENTITY_SEP: str = '/'#
__init__(db: Datalayer, on: Table, stop_event: Event, identifier: str = '', timeout: float | None = None, strategy: PollingStrategy | LogBasedStrategy | None = None)[source]#

__init__.

Parameters:
  • db – It is a superduperdb instance.

  • on – It is used to define a Collection on which CDC would be performed.

  • stop_event – A threading event flag to notify for stoppage.

  • identifier – A identifier to represent the listener service.

  • strategy

    Used to select strategy used for listening changes options:

    PollingStrategy (This strategy polls table every

    frequency seconds, more info at superduperdb.cdc.cdc.PollingStrategy)

    LogBasedStrategy (Not implemented yet)

listen() None[source]#
next_cdc(stream) None[source]#

Get the next stream of change observed on the given Collection.

on_create(ids: Sequence, db: Datalayer, table: Table) None[source]#

on_create. A helper on create event handler which handles inserted document in the change stream. It basically extracts the change document and build the taskflow graph to execute.

Parameters:
  • ids – Changed row ids.

  • db – a superduperdb instance.

  • table – The table on which change was observed.

on_delete(ids: Sequence, db: Datalayer, table: Table) None[source]#
on_update(ids: Sequence, db: Datalayer, table: Table) None[source]#
running() bool[source]#
setup_cdc()[source]#

Setup cdc change stream from user provided

stop() None[source]#

Stop listening cdc changes. This stops the corresponding services as well.

class superduperdb.backends.ibis.cdc.listener.PollingStrategyIbis(db: Datalayer, table: Table, strategy, primary_id: str = 'id')[source]#

Bases: object

fetch_ids()[source]#
get_strategy()[source]#
post_handling()[source]#
class superduperdb.backends.ibis.cdc.listener.PollingStrategyIbisByID(db: Datalayer, table: Table, strategy, primary_id: str = 'id')[source]#

Bases: PollingStrategyIbis

class superduperdb.backends.ibis.cdc.listener.PollingStrategyIbisByIncrement(db: Datalayer, table: Table, strategy, primary_id: str = 'id')[source]#

Bases: PollingStrategyIbis

fetch_ids()[source]#

Module contents#