Bar Synchronization

The lack of a standard formula in the literature and/or industry is not the problem, because the problem can actually be summarized as:

  • Bar Synchronization

Ticket #23 raises some questions as to whether backtrader can look into calculating a RelativeVolume indicator.

The requester needs to compare the volume of a given moment in time against the same moment in time on the previous trading day. Including:

  • Some pre-market data of unknown length

Having such a requirement invalidates the basic principle on which most indicators are built:

  • Having a fixed period which is used to look backwards

Furthermore and given the comparison is done intraday, something else has to be taken into account:

  • Some of the “intraday” instants may be missing (be it minutes or seconds)

    It is unlikely a data source will be missing a daily bar, but missing a minute or second bar is not that uncommon.

    The main reason being that there may have not been any negotiation at all. Or they might have been a problem at the negotiation exchange which actually prevented the bar from being recorded at all.

Taking into account all the aforementioned points some conclusions for the development of the indicator:

  • The period is not a period in this case but a buffer to make sure enough bars will be there to have the indicator kick in as soon as possible

  • Some bars may be missing

  • The main issue is synchronization

Luckily there is a key which comes to the rescue to aid with the synchronization:

  • Compared bars are “intraday” and hence counting the already seen days and the number of seen “bars” for a given moment of time enables synchronization

The previous day values are kept in a dictionary, because the “lookback” period as explained before is unknown.

Some other early ideas can be discarded, like for example implementing a DataFilter data source because this would actually bring the data source out of sync with other parts of the systme by removing the pre-market data. And the synchronization problem would also be there.

An idea to be explored would be creating a DataFiller which would fill in the missing minutes/seconds by using the last closing price and setting the volume to 0.

Getting hands on has also proven good to identify some extra needs in backtrader like a time2num function (an addition to the date2num and num2date family) and what will become extra methods for the lines:

  • Extracting the “day” and “fraction” (time) of day parts from the floating point representation of the day

    To be called “dt” and “tm”

In the meantime the code of the RelativeVolumeByBar indicator is presented below. Having the”period”/”buffer” calculation inside the indicator is not the preferred pattern, but it serves the purpose in this case.

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import collections
import datetime
import math


import backtrader as bt


def time2num(tm):
    """
    Convert :mod:`time` to the to the preserving hours, minutes, seconds
    and microseconds.  Return value is a :func:`float`.
    """
    HOURS_PER_DAY = 24.0
    MINUTES_PER_HOUR = 60.0
    SECONDS_PER_MINUTE = 60.0
    MUSECONDS_PER_SECOND = 1e6
    MINUTES_PER_DAY = MINUTES_PER_HOUR * HOURS_PER_DAY
    SECONDS_PER_DAY = SECONDS_PER_MINUTE * MINUTES_PER_DAY
    MUSECONDS_PER_DAY = MUSECONDS_PER_SECOND * SECONDS_PER_DAY

    tm_num = (tm.hour / HOURS_PER_DAY +
              tm.minute / MINUTES_PER_DAY +
              tm.second / SECONDS_PER_DAY +
              tm.microsecond / MUSECONDS_PER_DAY)

    return tm_num


def dtime_dt(dt):
    return math.trunc(dt)


def dtime_tm(dt):
    return math.modf(dt)[0]


class RelativeVolumeByBar(bt.Indicator):
    alias = ('RVBB',)
    lines = ('rvbb',)

    params = (
        ('prestart', datetime.time(8, 00)),
        ('start', datetime.time(9, 10)),
        ('end', datetime.time(17, 15)),
    )

    def _plotlabel(self):
        plabels = []
        for name, value in self.params._getitems():
            plabels.append('%s: %s' % (name, value.strftime('%H:%M')))

        return plabels

    def __init__(self):
        # Inform the platform about the minimum period needs
        minbuffer = self._calcbuffer()
        self.addminperiod(minbuffer)

        # Structures/variable to keep synchronization
        self.pvol = dict()
        self.vcount = collections.defaultdict(int)

        self.days = 0
        self.dtlast = 0

        # Keep the start/end times in numeric format for comparison
        self.start = time2num(self.p.start)
        self.end = time2num(self.p.end)

        # Done after calc to ensure coop inheritance and composition work
        super(RelativeVolumeByBar, self).__init__()

    def _barisvalid(self, tm):
        return self.start <= tm <= self.end

    def _daycount(self):
        dt = dtime_dt(self.data.datetime[0])
        if dt > self.dtlast:
            self.days += 1
            self.dtlast = dt

    def prenext(self):
        self._daycount()

        tm = dtime_tm(self.data.datetime[0])
        if self._barisvalid(tm):
            self.pvol[tm] = self.data.volume[0]
            self.vcount[tm] += 1

    def next(self):
        self._daycount()

        tm = dtime_tm(self.data.datetime[0])
        if not self._barisvalid(tm):
            return

        # Record the "minute/second" of this day has been seen
        self.vcount[tm] += 1

        # Get the bar's volume
        vol = self.data.volume[0]

        # If number of days is right, we saw the same "minute/second" last day
        if self.vcount[tm] == self.days:
            self.lines.rvbb[0] = vol / self.pvol[tm]

        # Synchronize the days and volume count for next cycle
        self.vcount[tm] = self.days

        # Record the volume for this bar for next cycle
        self.pvol[tm] = vol

    def _calcbuffer(self):
        # Period calculation
        minend = self.p.end.hour * 60 + self.p.end.minute
        # minstart = session_start.hour * 60 + session_start.minute
        # use prestart to account for market_data
        minstart = self.p.prestart.hour * 60 + self.p.prestart.minute

        minbuffer = minend - minstart

        tframe = self.data._timeframe
        tcomp = self.data._compression

        if tframe == bt.TimeFrame.Seconds:
            minbuffer = (minperiod * 60)

        minbuffer = (minbuffer // tcomp) + tcomp

        return minbuffer

Invoked through an script, which can be used as follows:

$ ./relative-volume.py --help
usage: relative-volume.py [-h] [--data DATA] [--prestart PRESTART]
                          [--start START] [--end END] [--fromdate FROMDATE]
                          [--todate TODATE] [--writer] [--wrcsv] [--plot]
                          [--numfigs NUMFIGS]

MultiData Strategy

optional arguments:
  -h, --help            show this help message and exit
  --data DATA, -d DATA  data to add to the system
  --prestart PRESTART   Start time for the Session Filter
  --start START         Start time for the Session Filter
  --end END, -te END    End time for the Session Filter
  --fromdate FROMDATE, -f FROMDATE
                        Starting date in YYYY-MM-DD format
  --todate TODATE, -t TODATE
                        Starting date in YYYY-MM-DD format
  --writer, -w          Add a writer to cerebro
  --wrcsv, -wc          Enable CSV Output in the writer
  --plot, -p            Plot the read data
  --numfigs NUMFIGS, -n NUMFIGS
                        Plot using numfigs figures

A test invocation:

$ ./relative-volume.py --plot

Generates this chart:

!image

The code for the script.

from __future__ import (absolute_import, division, print_function,
                        unicode_literals)

import argparse
import datetime

# The above could be sent to an independent module
import backtrader as bt
import backtrader.feeds as btfeeds

from relvolbybar import RelativeVolumeByBar

def runstrategy():
    args = parse_args()

    # Create a cerebro
    cerebro = bt.Cerebro()

    # Get the dates from the args
    fromdate = datetime.datetime.strptime(args.fromdate, '%Y-%m-%d')
    todate = datetime.datetime.strptime(args.todate, '%Y-%m-%d')

    # Create the 1st data
    data = btfeeds.VChartCSVData(
        dataname=args.data,
        fromdate=fromdate,
        todate=todate,
        )

    # Add the 1st data to cerebro
    cerebro.adddata(data)

    # Add an empty strategy
    cerebro.addstrategy(bt.Strategy)

    # Get the session times to pass them to the indicator
    prestart = datetime.datetime.strptime(args.prestart, '%H:%M')
    start = datetime.datetime.strptime(args.start, '%H:%M')
    end = datetime.datetime.strptime(args.end, '%H:%M')

    # Add the Relative volume indicator
    cerebro.addindicator(RelativeVolumeByBar,
                         prestart=prestart, start=start, end=end)

    # Add a writer with CSV
    if args.writer:
        cerebro.addwriter(bt.WriterFile, csv=args.wrcsv)

    # And run it
    cerebro.run(stdstats=False)

    # Plot if requested
    if args.plot:
        cerebro.plot(numfigs=args.numfigs, volume=True)


def parse_args():
    parser = argparse.ArgumentParser(description='MultiData Strategy')

    parser.add_argument('--data', '-d',
                        default='../../datas/2006-01-02-volume-min-001.txt',
                        help='data to add to the system')

    parser.add_argument('--prestart',
                        default='08:00',
                        help='Start time for the Session Filter')

    parser.add_argument('--start',
                        default='09:15',
                        help='Start time for the Session Filter')

    parser.add_argument('--end', '-te',
                        default='17:15',
                        help='End time for the Session Filter')

    parser.add_argument('--fromdate', '-f',
                        default='2006-01-01',
                        help='Starting date in YYYY-MM-DD format')

    parser.add_argument('--todate', '-t',
                        default='2006-12-31',
                        help='Starting date in YYYY-MM-DD format')

    parser.add_argument('--writer', '-w', action='store_true',
                        help='Add a writer to cerebro')

    parser.add_argument('--wrcsv', '-wc', action='store_true',
                        help='Enable CSV Output in the writer')

    parser.add_argument('--plot', '-p', action='store_true',
                        help='Plot the read data')

    parser.add_argument('--numfigs', '-n', default=1,
                        help='Plot using numfigs figures')

    return parser.parse_args()


if __name__ == '__main__':
    runstrategy()