hikari.iterators#
Lazy iterators for data that requires repeated API calls to retrieve.
For consumers of this API, the only class you need to worry about is
LazyIterator
. Everything else is internal detail only exposed for people who
wish to extend this API further!
Module Contents#
- class hikari.iterators.All(conditions)[source]#
Bases:
Generic
[ValueT
]Helper that wraps predicates and invokes them together.
Calling this object will pass the input item to each item, returning
True
only when all wrapped predicates return True when called with the given item.For example…
if w(foo) and x(foo) and y(foo) and z(foo): ...
is equivalent to
condition = All([w, x, y, z]) if condition(foo): ...
This behaves like a lazy wrapper implementation of the
all
builtin.Note
Like the rest of the standard library, this is a short-circuiting operation. This means that if a predicate returns
False
, no predicates after this are invoked, as the result is already known. In this sense, they are invoked in-order.Warning
You should not generally need to use this outside of extending the iterators API in this library!
- Parameters:
- conditions
typing.Callable
[[ValueT
], bool] The predicates to wrap.
- conditions
- class hikari.iterators.AttrComparator(attr_name, expected_value, cast=None)[source]#
Bases:
Generic
[ValueT
]A comparator that compares the result of a call with something else.
This uses the
spel
module internally.- Parameters:
- attr_name
str
The attribute name. Can be prepended with a
.
optionally. If the attribute name ends with a()
, then the call is invoked rather than treated as a property (useful for methods likestr.isupper
, for example).- expected_value
typing.Any
The expected value.
- cast
typing.Optional
[typing.Callable
[[ValueT
],typing.Any
]] Optional cast to perform on the input value when being called before comparing it to the expected value but after accessing the attribute.
- attr_name
- class hikari.iterators.BufferedLazyIterator[source]#
Bases:
Generic
[ValueT
],LazyIterator
[ValueT
],abc.ABC
A special kind of lazy iterator that is used by internal components.
The purpose of this is to provide an interface to lazily deserialize collections of payloads received from paginated API endpoints such as
GET /channels/{channel_id}/messages
, which will return a certain number of messages at a time on a low level. This class provides the base interface for handling lazily decoding each item in those responses and returning them in the expected format when iterating across this object.Implementations are expected to provide a
_next_chunk
private method which when awaited returns a lazy generator of each deserialized object to later yield. This will be iterated across lazily by this implementation, thus reducing the amount of work needed if only a few objects out of, say, 100, need to be deserialized.This
_next_chunk
should returnNone
once the end of all items has been reached.An example would look like the following:
async def some_http_call(i): ... class SomeEndpointLazyIterator(BufferedLazyIterator[SomeObject]): def __init__(self): super().__init__() self._i = 0 def _next_chunk(self) -> typing.Optional[typing.Generator[ValueT, None, None]]: raw_items = await some_http_call(self._i) self._i += 1 if not raw_items: return None generator = (SomeObject(raw_item) for raw_item in raw_items) return generator
- class hikari.iterators.FlatLazyIterator(values)[source]#
Bases:
Generic
[ValueT
],LazyIterator
[ValueT
]A lazy iterator that has all items in-memory and ready.
This can be iterated across as a normal iterator, or as an async iterator.
- class hikari.iterators.LazyIterator[source]#
Bases:
Generic
[ValueT
],abc.ABC
A set of results that are fetched asynchronously from the API as needed.
This is a
typing.AsyncIterable
andtyping.AsyncIterator
with several additional helpful methods provided for convenience.Examples
You can use this in multiple ways.
As an async iterable:
>>> async for item in paginated_results: ... process(item)
As an eagerly retrieved set of results (performs all API calls at once, which may be slow for large sets of data):
>>> results = await paginated_results >>> # ... which is equivalent to this... >>> results = [item async for item in paginated_results]
As an async iterator (not recommended):
>>> try: ... while True: ... process(await paginated_results.__anext__()) ... except StopAsyncIteration: ... pass
Additionally, you can make use of some of the provided helper methods on this class to perform basic operations easily.
Iterating across the items with indexes (like
enumerate
for normal iterables):>>> async for i, item in paginated_results.enumerate(): ... print(i, item) (0, foo) (1, bar) (2, baz)
Limiting the number of results you iterate across:
>>> async for item in paginated_results.limit(3): ... process(item)
- awaiting(window_size=10)[source]#
Await each item concurrently in a fixed size window.
Warning
Setting a large window size, or setting it to 0 to await everything is a dangerous thing to do if you are making API calls. Some endpoints will get ratelimited and cause a backup of waiting tasks, others may begin to spam global rate limits instead (the
fetch_user
endpoint seems to be notorious for doing this).Note
This call assumes that the iterator contains awaitable values as input. MyPy cannot detect this nicely, so any cast is forced internally. If the item is not awaitable, you will receive a
TypeError
instead. You have been warned. You cannot escape the ways of the duck type young grasshopper.- Parameters:
- window_size
int
The window size of how many tasks to await at once. You can set this to
0
to await everything at once, but see the below warning.
- window_size
- Returns:
LazyIterator
[ValueT
]The new lazy iterator to return.
- chunk(chunk_size)[source]#
Return results in chunks of up to
chunk_size
amount of entries.- Parameters:
- chunk_size
int
The limit for how many results should be returned in each chunk.
- chunk_size
- Returns:
LazyIterator
[typing.Sequence
[ValueT
]]LazyIterator
that emits each chunked sequence.
- async collect(collector)[source]#
Collect the results into a given type and return it.
- Parameters:
- collector
A function that consumes a sequence of values and returns a collection.
- enumerate(*, start=0)[source]#
Enumerate the paginated results lazily.
This behaves as an asyncio-friendly version of
enumerate
which uses much less memory than collecting all the results first and callingenumerate
across them.- Parameters:
- start
int
Optional int to start at. If omitted, this is
0
.
- start
- Returns:
LazyIterator
[typing.Tuple
[int
,T
]]A paginated results view that asynchronously yields an increasing counter in a tuple with each result, lazily.
Examples
>>> async for i, item in paginated_results.enumerate(): ... print(i, item) (0, foo) (1, bar) (2, baz) (3, bork) (4, qux) >>> async for i, item in paginated_results.enumerate(start=9): ... print(i, item) (9, foo) (10, bar) (11, baz) (12, bork) (13, qux) >>> async for i, item in paginated_results.enumerate(start=9).limit(3): ... print(i, item) (9, foo) (10, bar) (11, baz)
- filter(*predicates, **attrs)[source]#
Filter the items by one or more conditions.
Each condition is treated as a predicate, being called with each item that this iterator would return when it is requested.
All conditions must evaluate to
True
for the item to be returned. If this is not met, then the item is discarded and ignored, the next matching item will be returned instead, if there is one.- Parameters:
- *predicates
typing.Union
[typing.Callable
[[ValueT
], bool],typing.Tuple
[str
,typing.Any
]] Predicates to invoke. These are functions that take a value and return
True
if it is of interest, orFalse
otherwise. These may instead include 2-tuple
objects consisting of astr
attribute name (nested attributes are referred to using the.
operator), and values to compare for equality. This allows you to specify conditions such asmembers.filter(("user.bot", True))
.- **attrs
typing.Any
Alternative to passing 2-tuples. Cannot specify nested attributes using this method.
- *predicates
- Returns:
LazyIterator
[ValueT
]LazyIterator
that only emits values where all conditions are matched.
- flat_map(flattener)[source]#
Perform a flat mapping operation.
This will pass each item in the iterator to the given
function
parameter, expecting a newtyping.Iterable
ortyping.AsyncIterator
to be returned as the result. This means you can map to a newLazyIterator
,typing.AsyncIterator
,typing.Iterable
, async generator, or generator.Remember that
typing.Iterator
implicitly providestyping.Iterable
compatibility.This is used to provide lazy conversions, and can be used to implement reactive-like pipelines if desired.
All results are combined into one large lazy iterator and yielded lazily.
- Parameters:
- flattener
A function that returns either an async iterator or iterator of new values. Could be an attribute name instead.
- Returns:
LazyIterator
[AnotherValueT
]The new lazy iterator to return.
Examples
The following example generates a distinct collection of all mentioned users in the given channel from the past 500 messages.
def iter_mentioned_users(message: hikari.Message) -> typing.Iterable[Snowflake]: for match in re.findall(r"<@!?(\d+)>", message.content): yield Snowflake(match) mentioned_users = await ( channel .history() .limit(500) .map(".content") .flat_map(iter_mentioned_users) .distinct() )
- async last()[source]#
Return the last element of this iterator only.
Note
This method will consume the whole iterator if run.
- Returns:
ValueT
The last result.
- Raises:
LookupError
If no result exists.
- limit(limit)[source]#
Limit the number of items you receive from this async iterator.
- Parameters:
- limit
int
The number of items to get. This must be greater than zero.
- limit
- Returns:
LazyIterator
[ValueT
]A paginated results view that asynchronously yields a maximum of the given number of items before completing.
Examples
>>> async for item in paginated_results.limit(3): ... print(item)
- map(transformation)[source]#
Map the values to a different value.
- Parameters:
- transformation
typing.Union
[typing.Callable
[[ValueT
], bool],str
] The function to use to map the attribute. This may alternatively be a string attribute name to replace the input value with. You can provide nested attributes using the
.
operator.
- transformation
- Returns:
LazyIterator
[AnotherValueT
]LazyIterator
that maps each value to another value.
- async next()[source]#
Return the next element of this iterator only.
- Returns:
ValueT
The next result.
- Raises:
LookupError
If no more results exist.
- reversed()[source]#
Return a lazy iterator of the remainder of this iterator’s values reversed.
- Returns:
LazyIterator
[ValueT
]The lazy iterator of this iterator’s remaining values reversed.
- skip(number)[source]#
Drop the given number of items, then yield anything after.
- Parameters:
- number
int
The max number of items to drop before any items are yielded.
- number
- Returns:
LazyIterator
[ValueT
]A paginated results view that asynchronously yields all items AFTER the given number of items are discarded first.
- skip_until(*predicates, **attrs)[source]#
Discard items while all conditions are False.
Items after this will be yielded as normal.
- Parameters:
- *predicates
typing.Union
[typing.Callable
[[ValueT
], bool],typing.Tuple
[str
,typing.Any
]] Predicates to invoke. These are functions that take a value and return
True
if it is of interest, orFalse
otherwise. These may instead include 2-tuple
objects consisting of astr
attribute name (nested attributes are referred to using the.
operator), and values to compare for equality. This allows you to specify conditions such asmembers.skip_until(("user.bot", True))
.- **attrs
typing.Any
Alternative to passing 2-tuples. Cannot specify nested attributes using this method.
- *predicates
- Returns:
LazyIterator
[ValueT
]LazyIterator that only emits values once a condition has failed. All items before this are discarded.
- skip_while(*predicates, **attrs)[source]#
Discard items while all conditions are True.
Items after this will be yielded as normal.
- Parameters:
- *predicates
typing.Union
[typing.Callable
[[ValueT
], bool],typing.Tuple
[str
,typing.Any
]] Predicates to invoke. These are functions that take a value and return
True
if it is of interest, orFalse
otherwise. These may instead include 2-tuple
objects consisting of astr
attribute name (nested attributes are referred to using the.
operator), and values to compare for equality. This allows you to specify conditions such asmembers.skip_while(("user.bot", True))
.- **attrs
typing.Any
Alternative to passing 2-tuples. Cannot specify nested attributes using this method.
- *predicates
- Returns:
LazyIterator
[ValueT
]LazyIterator that only emits values once a condition has been met. All items before this are discarded.
- async sort(*, key=None, reverse=False)[source]#
Collect all results, then sort the collection before returning it.
- take_until(*predicates, **attrs)[source]#
Return each item until any conditions pass or the end is reached.
- Parameters:
- *predicates
typing.Union
[typing.Callable
[[ValueT
], bool],typing.Tuple
[str
,typing.Any
]] Predicates to invoke. These are functions that take a value and return
True
if it is of interest, orFalse
otherwise. These may instead include 2-tuple
objects consisting of astr
attribute name (nested attributes are referred to using the.
operator), and values to compare for equality. This allows you to specify conditions such asmembers.take_until(("user.bot", True))
.- **attrs
typing.Any
Alternative to passing 2-tuples. Cannot specify nested attributes using this method.
- *predicates
- Returns:
LazyIterator
[ValueT
]LazyIterator that only emits values until any conditions are matched.
- take_while(*predicates, **attrs)[source]#
Return each item until any conditions fail or the end is reached.
- Parameters:
- *predicates
typing.Union
[typing.Callable
[[ValueT
], bool],typing.Tuple
[str
,typing.Any
]] Predicates to invoke. These are functions that take a value and return
True
if it is of interest, orFalse
otherwise. These may instead include 2-tuple
objects consisting of astr
attribute name (nested attributes are referred to using the.
operator), and values to compare for equality. This allows you to specify conditions such asmembers.take_while(("user.bot", True))
.- **attrs
typing.Any
Alternative to passing 2-tuples. Cannot specify nested attributes using this method.
- *predicates
- Returns:
LazyIterator
[ValueT
]LazyIterator that only emits values until any conditions are not matched.