Skip to content

Commit 813e199

Browse files
committed
Added Throttle documentation.
1 parent a4aede4 commit 813e199

File tree

4 files changed

+22
-10
lines changed

4 files changed

+22
-10
lines changed

.travis.yml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,6 @@
11
notifications:
22
email: false
33

4-
sudo: false
5-
64
language: php
75

86
php:

README.md

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Porter <img src="https://github.com/ScriptFUSION/Porter/blob/master/docs/images/
1111

1212
Porter is the all-purpose PHP data importer. She fetches data from anywhere and serves it as a single record or an iterable [record collection](#record-collections), encouraging processing one record at a time instead of loading full data sets into memory at once. Her [durability](#durability) feature provides automatic, transparent recovery from intermittent network connectivity errors by default.
1313

14-
Porter's interface trichotomy of [providers](#providers), [resources](#resources) and [connectors](#connectors) maps well to APIs. For example, a typical API such as GitHub would define the provider as GitHub, a resource as `GetUser` or `ListRepositories` and the connector could be [HttpConnector][].
14+
Porter's [interface trichotomy](#overview) of [providers](#providers), [resources](#resources) and [connectors](#connectors) maps well to APIs. For example, a typical API such as GitHub would define the provider as GitHub, a resource as `GetUser` or `ListRepositories` and the connector could be [HttpConnector][].
1515

1616
Porter provides a dual API for synchronous and [asynchronous](#asynchronous) imports, both of which are concurrency safe, so multiple imports can be paused and resumed simultaneously. Asynchronous mode allows large scale imports across multiple connections to work at maximum efficiency without waiting for each network call to complete.
1717

@@ -149,8 +149,7 @@ Options may be configured using the methods below.
149149
- `enableCache()` &ndash; Enables caching. Requires a `CachingConnector`.
150150
- `setMaxFetchAttempts(int)` &ndash; Sets the maximum number of fetch attempts per connection before failure is considered permanent.
151151
- `setFetchExceptionHandler(FetchExceptionHandler)` &ndash; Sets the exception handler invoked each time a fetch attempt fails.
152-
153-
In synchronous code, import specifications are an instance of `ImportSpecification`
152+
- `setThrottle(Throttle)` &ndash; Sets the asynchronous connection throttle, invoked each time a connector fetches data. Applies to `AsyncImportSpecification` only.
154153

155154
Record collections
156155
------------------
@@ -195,11 +194,15 @@ Programming asynchronously requires an understanding of Amp, the async framework
195194

196195
### Throttling
197196

198-
The asynchronous import model is very powerful because it changes our application's performance from being I/O-bound to being CPU-bound. That is, in the traditional synchronous model, each import operation must wait for the previous to complete before the next begins, meaning the total import time depends how long it takes each import's network I/O to complete. In the async model, since we send many requests concurrently without waiting for the previous to complete, on average each import operation will only take as long as our CPU takes to process it, since we are busy processing another import during network latency.
197+
The asynchronous import model is very powerful because it changes our application's performance model from I/O-bound, limited by the speed of the network, to CPU-bound, limited by the speed of the CPU. In the traditional synchronous model, each import operation must wait for the previous to complete before the next begins, meaning the total import time depends on how long it takes each import's network I/O to finish. In the async model, since we send many requests concurrently without waiting for the previous to complete, on average each import operation only takes as long as our CPU takes to process it, since we are busy processing another import during network latency (except during the initial "spin-up").
198+
199+
Synchronously, we seldom trip protection measures even for high volume imports, however the naïve approach to asynchronous imports is often fraught with perils. If we import 10,000 HTTP resources at once, one of two things usually happens: either we run out of PHP memory and the process terminates prematurely or the HTTP server rejects us after sending too many requests in a short period. The solution is throttling.
199200

200-
High volume synchronous imports are, in a way, self-throttling and it is rare to trip protection measures in this mode, however the naïve approach to asynchronous imports is often fraught with perils. For example, when we import 10,000 HTTP resources at once, one of two things usually happens: either we run out of PHP memory and the process is killed or the HTTP server blocks us for sending too many requests in a short period. The solution is throttling.
201+
[Async Throttle][] is a library included with Porter to throttle asynchronous imports. The throttle works by preventing additional operations starting when too many are executing concurrently, based on user-defined limits. By default, `NullThrottle` is assigned, which does not throttle connections. `DualThrottle` can be used to set two independent connection rate limits: the maximum number of connections per second and the maximum number of concurrent connections. A `DualThrottle` can be assigned by modifying the import specification as follows.
201202

202-
We provide [Async Throttle][] to throttle asynchronous imports. The Async Throttle is a separate project that does not have any direct integration with Porter because that is not needed. The throttle operates on any Amp promises, such as those returned by Porter. The throttle works by preventing additional operations starting when too many are concurrently executing, based on user-defined limits.
203+
```php
204+
(new AsyncImportSpecification)->setThrottle(new DualThrottle)
205+
```
203206

204207
Transformers
205208
------------

composer.json

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,7 @@
2626
},
2727
"suggest" : {
2828
"connectors/http": "Provides an HTTP connector for Porter providers.",
29-
"transformers/mapping-transformer": "Transforms records using Mappings and provides sub-imports.",
30-
"async/throttle": "Limits throughput of asynchronous imports."
29+
"transformers/mapping-transformer": "Transforms records using Mappings and provides sub-imports."
3130
},
3231
"autoload": {
3332
"psr-4": {

src/Specification/AsyncImportSpecification.php

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,11 +60,23 @@ protected static function createDefaultRecoverableExceptionHandler(): Recoverabl
6060
return new ExponentialAsyncDelayRecoverableExceptionHandler;
6161
}
6262

63+
/**
64+
* Gets the asynchronous connection throttle, invoked each time a connector fetches data.
65+
*
66+
* @return Throttle Asynchronous connection throttle.
67+
*/
6368
final public function getThrottle(): Throttle
6469
{
6570
return $this->throttle ?? $this->throttle = new NullThrottle;
6671
}
6772

73+
/**
74+
* Sets the asynchronous connection throttle, invoked each time a connector fetches data.
75+
*
76+
* @param Throttle $throttle Asynchronous connection throttle.
77+
*
78+
* @return $this
79+
*/
6880
final public function setThrottle(Throttle $throttle): self
6981
{
7082
$this->throttle = $throttle;

0 commit comments

Comments
 (0)