Apache Kafka-inspired data stream buffer.
Data stream buffer.
Global offset within a Broker
.
Offset itself can live independently from its generating Broker
.
initialize : { numSegments : Basics.Int, segmentSize : Basics.Int } -> Broker a
Initializes a Broker
with a set of configuration.
numSegments
- Number of internal segments. 2 ≤ numSegments
≤ 100segmentSize
- Size of each segment. 100 ≤ segmentSize
≤ 100,000Constraints for these configurations are not yet studied well.
append : a -> Broker a -> Broker a
Append an item to a Broker.
read : Offset -> Broker a -> Maybe ( a, Offset )
Read a Broker
by supplying previously read Offset
(consumer offset),
returning a next item and its Offset
, or Nothing
if all items are consumed.
If the Offset
is too old and the target segment is already evicted, returns the oldest readable item.
Currently it assumes Brokers cannot be reconfigured.
This means, if the Offset
is produced from the same Broker
,
it can never overtake the current write pointer or become out of bound of the Broker
.
readOldest : Broker a -> Maybe ( a, Offset )
Read a Broker
from the oldest item. Returns an item and its Offset
,
or Nothing
if the Broker
is empty.
get : Offset -> Broker a -> Maybe a
Get an item exactly at an Offset
.
Returns Nothing
if target segment is already evicted or somehow invalid.
update : Offset -> (a -> a) -> Broker a -> Broker a
Update an item at an Offset
of a Broker
.
If target segment is already evicted or not-updatable (soon-to-be-evicted), the Broker
kept unchanged.
decoder : Json.Decode.Decoder a -> Json.Decode.Decoder (Broker a)
Decode JS value into Broker. You must supply Decoder for items.
Paired with encode
, you can "dump and reload" an exisiting Broker.
encode : (a -> Json.Encode.Value) -> Broker a -> Json.Encode.Value
Encode Broker into JS value. You must supply encode function for items.
Paired with decoder
, you can "dump and reload" an exisitng Broker.
Do note that, this function naively encodes internal structure of Broker into JS values, which may require non-ignorable amount of work (both in encode and decode) if capacity of the Broker is big. More sophisticated "resume" behavior might be needed later.
capacity : Broker a -> Basics.Int
Returns capacity (number of possible elements) of the Broker.
isEmpty : Broker a -> Basics.Bool
Returns whether a Broker is empty or not.
oldestReadableOffset : Broker a -> Maybe Offset
Returns the oldest readable Offset
of a Broker
. Items older than this Offset
are already evicted.
If the Broker
is yet empty, returns Nothing
.
nextOffsetToWrite : Broker a -> Offset
Returns an Offset
that next item will be written to.
offsetToString : Offset -> String
Converts an Offset
into a sortable String
representation.
Also usable when you need to store and reload consumer offset.
offsetFromString : String -> Maybe Offset
Tries to convert a String
into Offset
.
Can be used to reload consumer offset from external storage.
Make sure that a reloaded Offset
is used against the very Broker
that produced that Offset
.