❤
npm
Sign UpSign In

datalake

1.4.2 • Public • Published

Redis Data Lake

Use Redis as a document database with extensive search capabilities

Installation

USING NPM

$ npm install datalake --save

In node.js

const datalake = require('datalake');
const dl = new datalake();

Implementation

Code:

const dl = require('datalake');
const datalake = new datalake();
 
datalake.MethodName(InputParamater).
    then((response) => {
        // do your logics here
    }).
    catch((err) => {
        console.log(err);
    });

Available Methods

S.No API Name Method Required Fields Optional Fields Sample PostData Notes
2 /CloseConnection GET ------------ ------------ ------------ ------------
2 /ConfigureSearchIndex POST ShortCodes(sc, type, csv), Schema, Keyword ------------ /ConfigureSearchIndex ------------
2 /CreateBackupData POST Schema, Guid ------------ /CreateBackupData ------------
2 /CreateConnection POST host,dbname port,password,maxConnection /CreateConnection ------------
2 /Fetch POST
2 /GetAllSchemaData POST Schema Keyword, Guid /GetAllSchemaData This API cannot used in big data (Server will crash)
2 /GetCacheData POST Schema,Key ------------ /GetCacheData ------------
2 /getDistinctRecords POST
2 /InsertData POST Schema, Keyword, MetaData ------------ /InsertData ------------
2 /ListSchemas GET ------------ ------------ ------------ ------------
2 /ListSearchIndex POST Schema, Keyword ------------ /ListSearchIndex ------------
2 /RefreshSchemaSearchIndex POST Schema Guid /RefreshSchemaSearchIndex ------------
2 /RemoveData POST
2 /RestoreData POST Schema, Guid, Version ------------ /RestoreData ------------
2 /Search POST Schema, Keyword (SearchFields), recordFrom, recordUpto, Guid /Search Schema, (SearchFields) can be single value or can be comma (,) seperated or can be ranged (~). Either (SearchFields) or Guid should be given else This API cannot used in big data (Server will crash)
2 /SearchDataByProperty POST Schema, Keyword, (SearchFields) ------------ /SearchMultipleData (SearchFields) can be single value or can be comma (,) seperated or can be ranged (~)
2 /SearchMultipleData POST Schema, Keyword PropertyField, PropertyValue, Guid /SearchDataByProperty Schema, PropertyField, PropertyValue can be single value or can be comma (,) seperated or can be ranged (~). Either (SearchFields) or Guid should be given else This API cannot used in big data (Server will crash)
2 /SearchTopRecords POST Schema, Keyword, Nextbatch, BATCHCOUNT, field ------- /SearchTopRecords Nextbatch will be first element of response data
2 /SetCacheData POST Schema, Key, Data Timeout /SetCacheData If Timeout is not given, Data will be stored in redis permanently
2 /ShowConnectionStatus GET ------------ ------------ ------------ ------------

Sample Input

createConnection

POST : createConnection() → {undefined}

This will Create Redis Connection pool object.

Post Data:

{
    "host": "localhost",
    "dbname": "0",
    "port": "1607", // Optional
    "password": "YourPassword" // Optional
}

closeConnection

GET : closeConnection() → {undefined}

This will close already created Redis connection pool if any.

showStatus

GET : showStatus() → {JSON}

Returns the Status of Redis connection

Response:

{
  "Status": "TP Redis Module Loaded and Ready to Rock and Roll...",
  "Message": "DataLake - Licensed by SkunkworxLab, LLC."
}

SetupSearchHash

POST : SetupSearchHash() → {JSON}

Defining Redis Schema index keys

Post Data:

{
    "ShortCodes": [
        {
            "sc": "Name",
            "type": "string"
        },
        {
            "sc": "Email",
            "type": "string"
        },
        {
            "sc": "Phone",
            "type": "integer"
        },
        {
            "sc": "Joined",
            "type": "date"
        },
        {
            "sc": "WorkLocation",
            "type": "string",
            "csv": "true"
        }
    ],
    "Schema": "company",
    "Keyword": "employee"
}

InsertData

POST : InsertData() → {JSON}

This will insert Key value pair against the Schema given

Post Data:

{
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003", // optional
    "Schema": "company",
    "Keyword": "employee",
    "MetaData": "{ \"Name\": \"Aswin\",  \"Email\": \"aswin5010@gmail.com\", \"Age\": \"25\", \"Phone\": \"9876543210\", \"Joined\": \"06-March-2016\", \"WorkLocation\": [\"California\", \"India\"] }" // MetaData will be stringified json object.
}

GetAllSchemaData

POST : GetAllSchemaData() → {JSON}

This will search records for given Keyword, Schema and returns all Key-value pair available in the schema. Result can be filtered with particular Guid

Post Data:

{
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003", // optional
    "Schema": "companies",
    "Keyword": "employee"
}

SearchTidalPoolHash

POST : SearchTidalPoolHash() → {JSON}

This will search record for given PropertyField, PropertyValue, Keyword and SchemaName including comma(,) separated PropertyValues and range of PropertyValue

Implementation

Post Data:

{
    "Schema":"companies",
    "Keyword": "employee",
    "Name": "Aswin",
    "Email": "aswin5010@gmail.com",
    "Joined": "6-March-2018"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.SearchTidalPoolHash(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success",
  "TotalCount": 1,
  "items": [
    {
      "Guid": "ced293b4-23d8-490f-8944-9495bad5b003",
      "Keyword": "Label",
      "MetaData": "{ \"FirstName\": \"Aswin\",  \"City\": \"New York\", \"Age\": \"25\" }"
    }
  ]
}

GetDatafromSchemas

POST : GetDatafromSchemas() → {JSON}

This Method is the combination of GetSearchHashSchema, GetTidalPoolSchema, and SearchTidalPoolHash. This will search the records for given PropertyField, PropertyValue, Keyword and SchemaName. Multi-value search of records using comma(,) separated list of PropertyValue(s) or range between the PropertyValue(s) or it can be filtered with particular Guid. It can also performs multiple comma(,) separated Schema Name search.

Implementation

Post Data:

{
    "Schema": "Test",
    "PropertyField": ["FirstName"],
    "PropertyValue": ["Aswin"]
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetDatafromSchemas(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Message": "Success",
    "items": {
        "Test": [
            {
                "Guid": "ced293b4-23d8-490f-8944-9495bad5b003",
                "Keyword": "Label",
                "MetaData": "{ \"FirstName\": \"Aswin\",  \"City\": \"New York\", \"Age\": \"25\" }"
            }
        ]
    }
}

GetSchemaList

GET : GetSchemaList() → {undefined}

GetSchemaList will return list of Schema Names available in the Selected Redis DB.

Implementation

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetSchemaList(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Message": "Success",
    "items": [
        "Test"
    ]
}

GetSearchHashSchema

POST : GetSearchHashSchema() → {JSON}

This will return the defined Redis Schema index keys for the given Schema Name.

Implementation

Post Data:

{
    "VESShortCode": "Test",
    "Keyword": "Label"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetSearchHashSchema(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "VESShortCode": "Test",
  "Keyword": "Label",
  "SearchHash": [
    {
      "sc": "FirstName",
      "type": "string"
    },
    {
      "sc": "City",
      "type": "string"
    },
    {
      "sc": "Age",
      "type": "integer"
    }
  ]
}

GetTpSearchHash

POST : GetTpSearchHash() → {JSON}

This will return the defined Redis Schema index keys for the given Schema Name.

Implementation

Post Data:

{
    "Keyword": "Label",
    "VESShortCode": "Test"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetTpSearchHash(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success",
  "items": [
    {
      "sc": "FirstName",
      "type": "string"
    },
    {
      "sc": "City",
      "type": "string"
    },
    {
      "sc": "Age",
      "type": "integer"
    }
  ]
}

SetKeyData

POST : SetKeyData() → {JSON}

Set cache data into the Redis DB for the given key-value pair with TTL(Time to live) provided. If TTL(Time to live) not provided, it will set cache data as permanent data.

Implementation

Post Data:

{
    "Schema": "doshArnCache:12345",
    "Key": "Data",
    "Data": "{ \"FirstName\": \"Aswin\",  \"City\": \"New York\", \"Age\": \"25\" }",
    "TimeOut": 100
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.SetKeyData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success"
}

GetKeyData

POST : GetKeyData() → {JSON}

Get the value from cache memory for given search key before the TTL(Time to live) expire

Implementation

Post Data:

{
    "Schema": "doshArnCache:12345",
    "Key": "Data"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.GetKeyData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Data": [
        "{ \"FirstName\": \"Aswin\",  \"City\": \"New York\", \"Age\": \"25\" }"
    ]
}

RefreshTidalPoolData

POST : RefreshTidalPoolData() → {JSON}

Refresh/Re-Build all the search index for the given Schema Name

Implementation

Post Data:

{
    "VESShortCode": "Test"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.RefreshTidalPoolData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Message": "Success"
}

UpdateTidalPoolHash

POST : UpdateTidalPoolHash() → {JSON}

Refresh/Re-Build particular search index for the given Guid, Schema Name.

Implementation

Post Data:

{
    "VESShortCode": "Test",
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.UpdateTidalPoolHash(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success"
}

SnapshotTidalPoolData

POST : SnapshotTidalPoolData() → {JSON}

It Creates Backup copy of given Guid, Schema Name.

Implementation

Post Data:

{
    "VESShortCode": "Test",
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.SnapshotTidalPoolData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
    "Status": "true",
    "Message": "Success",
    "Version": "1"
}

RemoveTidalPoolData

POST : RemoveTidalPoolData() → {JSON}

It Creates Backup copy of given Guid, Schema Name and Removes Key-value pair from the Schema.

Implementation

Post Data:

{
    "Keyword": "Label",
    "VESShortCode": "Test",
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.RemoveTidalPoolData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success"
}

RollbackTidalPoolData

POST : RollbackTidalPoolData() → {JSON}

It Creates Backup copy of current Key-Value pair for the Given Guid-Schema and Replaces old Key-Value pair from Archive.

Implementation

Post Data:

{
    "Version": "1",
    "VESShortCode": "Test",
    "Guid": "ced293b4-23d8-490f-8944-9495bad5b003"
}

Code:

const datalake = require('datalake');
const dl = new datalake();
dl.RollbackTidalPoolData(postData).
    then((response) => {
        // do your logics here
        console.log(response);
    }).
    catch((err) => {
        console.log(err);
    });

Response:

{
  "Status": "true",
  "Message": "Success"
}

Readme

Keywords

Package Sidebar

Install

npm i datalake

Repository

github.com/madskunker/datalake

Homepage

github.com/madskunker/datalake#readme

Weekly Downloads

1

Version

1.4.2

License

ISC

Unpacked Size

147 kB

Total Files

4

Last publish

Collaborators

  • madskunker
  • rajpraba
  • aswin1054
Try on RunKit
Report malware