Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
c8d74d7
GPII-1987: Initial check in of a start on creating data loaders.
cindyli Nov 11, 2016
ef671e4
GPII-1987: In progress with dataLoader.js
cindyli Nov 14, 2016
3c0c6ca
GPII-1987: In progress of implementing data loaders.
cindyli Nov 16, 2016
64c175e
GPII-1987: In progress of loading the authorization data.
cindyli Nov 21, 2016
83a426a
Merge branch 'master' into GPII-1987
cindyli Nov 25, 2016
c4cd5b4
Merge branch 'master' into GPII-1987
cindyli Dec 7, 2016
22e41d1
GPII-1987: Dropped the use of kettle.dataSource.CouchDB by using kett…
cindyli Dec 9, 2016
5f7285d
Merge branch 'GPII-1987' of https://github.com/cindyli/universal into…
cindyli Dec 15, 2016
de78b86
GPII-1987: Completed the script for loading authorization data into C…
cindyli Dec 16, 2016
19b8436
GPII-1987: In progress of creating the preferences data loader.
cindyli Dec 19, 2016
a27423b
GPII-1987: In progress of creating preferences data loader.
cindyli Dec 21, 2016
a06801d
GPII-1987: A little adjustment on constructing the preferences data s…
cindyli Dec 21, 2016
745bbfc
GPII-1987: Completed on the preferences data loader.
cindyli Dec 23, 2016
23762e9
GPII-1987: Allow to pass the data base name as an option. Added code …
cindyli Jan 3, 2017
6ce660c
GPII-1987: Splitted authDataLoader.js into 2 js files that one contai…
cindyli Jan 3, 2017
ec33391
Merge branch 'master' into GPII-1987
cindyli Jan 24, 2017
7798a42
GPII-1987: Refactored data loaders as well as adding test for the aut…
cindyli Jan 31, 2017
eef277d
GPII-1987: Added the validation process before the loading starts.
cindyli Feb 2, 2017
1bd667d
GPII-1987: File renaming and tests fixing.
cindyli Feb 2, 2017
cf907b9
GPII-1987: Adjusted auth data loader tests.
cindyli Feb 3, 2017
47764a8
GPII-1987: Refactored auth data loader test to reuse the code for tes…
cindyli Feb 6, 2017
908f9ff
GPII-1987: Added error test case for auth data loader.
cindyli Feb 7, 2017
e036123
GPII-1987: Created "gpii.dataLoader.reporter" grade to report success…
cindyli Feb 7, 2017
89c14cf
GPII-1987: Added tests for preferences data loader.
cindyli Feb 9, 2017
86bb666
GPII-1987: Corrected a duplicate grade name that causes running all-t…
cindyli Feb 9, 2017
0b0a07f
Merge branch 'master' into GPII-1987
cindyli Feb 9, 2017
c88e17c
GPII-1987: Added README for "scripts" directory.
cindyli Feb 10, 2017
521f72c
GPII-1987: In progress of using data loaders to set up GPII productio…
cindyli Feb 13, 2017
025cba4
GPII-1987: Uses avtar's fix on ansible-preferences-server that allows…
cindyli Feb 15, 2017
67715b9
GPII-1987: Addressed code review comments.
cindyli Feb 17, 2017
0e75218
GPII-1987: Adjusted data loader scripts to use the new gpii dataLoade…
cindyli Feb 17, 2017
1af51a9
GPII-1987: Use the correct branch to pull out ansible-preferences-ser…
cindyli Feb 17, 2017
72aab68
GPII-1987: Use the latest gpii-pouchdb changes to include the improve…
cindyli Feb 21, 2017
34e211e
GPII-1987: Sync up infusion versions used by universal and gpii-pouch…
cindyli Feb 21, 2017
4f7f072
GPII-1987: Use the correct gpii-pouchdb version after checking with T…
cindyli Feb 22, 2017
3907f0c
GPII-1987: Improved code comments and linted.
cindyli Feb 23, 2017
24b45c3
Merge branch 'GPII-1987' of https://github.com/cindyli/universal into…
cindyli Feb 23, 2017
2818200
GPII-1987: Switched to use gpii-ops/ansible-preferences-server master…
cindyli Feb 24, 2017
8d03025
Merge branch 'master' into GPII-1987
cindyli Mar 1, 2017
2778b60
Merge branch 'master' into GPII-1987
cindyli Apr 26, 2017
f6a1d79
GPII-1987: Experimenting with the lastest gpii-pouchdb release.
cindyli Apr 26, 2017
2da973e
GPII-1987: Updated to use the latest gpii-pouchdb repo commits.
cindyli Apr 28, 2017
f02e7d8
GPII-1987: Updated to use the latest gpii-pouchdb release.
cindyli May 1, 2017
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 82 additions & 0 deletions documentation/DataLoader.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
## Data Loaders

Data loaders are used for setting up GPII production environment by loading authorization and/or preferences test data into CouchDB. See [Data Loader issue ticket](https://issues.gpii.net/browse/GPII-1987) for details.

### Preferences Data Loader

**Script Location:** `scripts/dataLoader-prefs.js`

**Infusion Component:** `gpii.dataLoader.prefsDataLoader`

**Component Source Code:** `gpii/node_modules/dataLoader/src/prefsDataLoader.js`

Preferences Data Loader reads all JSON files from the given directory. Each JSON file is loaded as a individual CouchDB document with its file name as the corresponding document id.

To start the data loading of the preferences data, run the following command in the `universal` directory:

```
node scripts/dataLoader-prefs.js
```

#### Component Options

| Option | Type | Description | Default |
| ----------------- | ---------- | ----------- | ------- |
| `dbName` | String | Required. The name of the CouchDB database to be created for loading preferences data. If the database already exists, delete it and recreate. | preferences |
| `dataPath` | String | Required. The directory where all preference JSON files reside. | %universal/testData/preferences/ |
| `couchDbUrl` | String | Required. The URL to the CouchDB. | http://localhost:5984 |

#### Supported Events

| Event | Description | Parameters | Parameters Description |
| ----- | ----------- | ---------- | ---------------------- |
| `onDataLoaded` | Fires when all preference data has been loaded into the CouchDB. | None | |
| `onDataLoadedError` | Fires when an error occurs at loading preference data. | errorMsg | The error message. |

### Authorization Data Loader

**Script Location:** `scripts/dataLoader-auth.js`

**Infusion Component:** `gpii.dataLoader.authDataLoader`

**Component Source Code:** `gpii/node_modules/dataLoader/src/authDataLoader.js`

Authorization Data Loader reads given JSON files from the file system and load them into CouchDB as they are.

To start the data loading of the authorization data, run the following command in the `universal` directory:

```
node scripts/dataLoader-auth.js
```

#### Component Options

| Option | Type | Description | Default |
| ----------------- | ---------- | ----------- | ------- |
| `dbName` | String | Required. The name of the CouchDB database to be created for loading authorization data. If the database already exists, delete it and recreate. | auth |
| `dataFile` | Array | Required. An array of JSON data files to be loaded. See the section below for an example of a JSON data file.| ["%universal/testData/security/TestOAuth2DataStore.json", "%universal/gpii/node_modules/gpii-oauth2/gpii-oauth2-datastore/dbViews/views.json"] |
| `couchDbUrl` | String | Required. The URL to the CouchDB. | `http://localhost:5984` |

**Example of A JSON Data File**
```
[
{
"_id": "user-1",
"type": "user",
"name": "chromehc",
"defaultGpiiToken": "review3_chrome_high_contrast"
}, {
"_id": "user-2",
"type": "user",
"name": "ma1",
"defaultGpiiToken": "review3_ma1"
}
]
```

#### Supported Events

| Event | Description | Parameters | Parameters Description |
| ----- | ----------- | ---------- | ---------------------- |
| `onDataLoaded` | Fires when all preference data has been loaded into the CouchDB. | None | |
| `onDataLoadedError` | Fires when an error occurs at loading preference data. | errorMsg | The error message. |
10 changes: 10 additions & 0 deletions gpii/node_modules/dataLoader/README.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions gpii/node_modules/dataLoader/index.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

48 changes: 48 additions & 0 deletions gpii/node_modules/dataLoader/src/authDataLoader.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading