# redis OM

> **Content description**: How to create a cluster of redis to perform fast and scalalble search in python.

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams. — Redis Website
(But i digress as it can be ***the only DB you need*** with super capabilites of hashes, Object mapping, Pub/Sub, Search, Indexing and json, Graph and timeseries, and most importantly speed and reliability)

Understanding redis
Every Redis Cluster node requires two TCP connections. The normal Redis TCP port used to serve clients, i.e. 6379, plus the port obtained by adding 10000 to the data port, so 16379. Make sure you open both ports in your firewall, otherwise Redis cluster nodes will be not able to communicate.

Redis Cluster does not use consistent hashing, but a different form of sharding where every key is conceptually part of what we call an hash slot. There are 16384 hash slots in Redis Cluster.

Every node in a Redis Cluster is responsible for a subset of the hash slots, so for example you may have a cluster with 3 nodes, where:

* Node A contains hash slots from 0 to 5500.
* Node B contains hash slots from 5501 to 11000.
* Node C contains hash slots from 11001 to 16383.

You can use docker & docker-compose to set up a local cluster vai using multiple templates on the web eg.: https://pierreabreu.medium.com/building-redis-cluster-with-docker-compose-9569ddb6414a

I wish to explore on the search features of Redis combined with the Json module for redis.

In the tutorial we'll also create a small microservice for extending and interacting with redis.

# env setup for redis om

$ export export REDIS_OM_URL=redis://<username>:<password>@<host>:<port>

# Using Pokemon data:

<summary>pokemon csv:</summary>

Name,Type_1,Type_2,Max_CP,Max_HP
Bulbasaur,Grass,Poison,1079,83
Ivysaur,Grass,Poison,1643,107
Venusaur,Grass,Poison,2598,138
Charmander,Fire,,962,73
Charmeleon,Fire,,1568,103
Charizard,Fire,Flying,2620,135
Squirtle,Water,,1015,81
Wartortle,Water,,1594,105
Blastoise,Water,,2560,137
Caterpie,Bug,,446,83
Metapod,Bug,,481,91
Butterfree,Bug,Flying,1465,107
Weedle,Bug,Poison,452,75
Kakuna,Bug,Poison,488,83
Beedrill,Bug,Poison,1450,115
Pidgey,Normal,Flying,684,75
Pidgeotto,Normal,Flying,1232,111
Pidgeot,Normal,Flying,2106,143
Rattata,Normal,,585,59
Raticate,Normal,,1454,99
Spearow,Normal,Flying,691,75
Fearow,Normal,Flying,1758,115
Ekans,Poison,,830,67
Arbok,Poison,,1779,107
Pikachu,Electric,,894,67
Raichu,Electric,,2042,107
Sandshrew,Ground,,804,91
Sandslash,Ground,,1823,130
Nidoran♀,Poison,,882,99
Nidorina,Poison,,1414,122
Nidoqueen,Poison,Ground,2502,154
Nidoran♂,Poison,,849,84
Nidorino,Poison,,1382,108
Nidoking,Poison,Ground,2492,140
Clefairy,Fairy,,1209,122
Clefable,Fairy,,2414,162
Vulpix,Fire,,837,72
Ninetales,Fire,,2203,127
Jigglypuff,Normal,Fairy,924,194
Wigglytuff,Normal,Fairy,2192,233
Zubat,Poison,Flying,647,75
Golbat,Poison,Flying,1935,130
Oddish,Grass,Poison,1156,83
Gloom,Grass,Poison,1701,107
Vileplume,Grass,Poison,2510,130
Paras,Bug,Grass,923,67
Parasect,Bug,Grass,1759,107
Venonat,Bug,Poison,1036,107
Venomoth,Bug,Poison,1903,122
Diglett,Ground,,460,27
Dugtrio,Ground,,1176,67
Meowth,Normal,,761,75
Persian,Normal,,1643,115
Psyduck,Water,,1117,91
Golduck,Water,,2403,138
Mankey,Fighting,,884,75
Primeape,Fighting,,1877,115
Growlithe,Fire,,1344,99
Arcanine,Fire,,3005,154
Poliwag,Water,,801,75
Poliwhirl,Water,,1350,115
Poliwrath,Water,Fighting,2523,154
Abra,Psychic,,604,51
Kadabra,Psychic,,1140,75
Alakazam,Psychic,,1826,99
Machop,Fighting,,1097,122
Machoke,Fighting,,1773,138
Machamp,Fighting,,2612,154
Bellsprout,Grass,Poison,1125,91
Weepinbell,Grass,Poison,1736,115
Victreebel,Grass,Poison,2548,138
Tentacool,Water,Poison,911,75
Tentacruel,Water,Poison,2236,138
Geodude,Rock,Ground,855,75
Graveler,Rock,Ground,1443,99
Golem,Rock,Ground,2319,138
Ponyta,Fire,,1526,91
Rapidash,Fire,,2215,115
Slowpoke,Water,Psychic,1227,154
Slowbro,Water,Psychic,2615,162
Magnemite,Electric,Steel,897,51
Magneton,Electric,Steel,1893,91
Farfetch'd,Normal,Flying,1272,94
Doduo,Normal,Flying,861,67
Dodrio,Normal,Flying,1849,107
Seel,Water,,1114,115
Dewgong,Water,Ice,2161,154
Grimer,Poison,,1293,138
Muk,Poison,,2621,178
Shellder,Water,,828,59
Cloyster,Water,Ice,2067,91
Gastly,Ghost,Poison,810,59
Haunter,Ghost,Poison,1390,83
Gengar,Ghost,Poison,2093,107
Onix,Rock,Ground,863,67
Drowzee,Psychic,,1082,107
Hypno,Psychic,,2199,146
Krabby,Water,,797,59
Kingler,Water,,1836,99
Voltorb,Electric,,845,75
Electrode,Electric,,1657,107
Exeggcute,Grass,Psychic,1107,107
Exeggutor,Grass,Psychic,2976,162
Cubone,Ground,,1013,91
Marowak,Ground,,1668,107
Hitmonlee,Fighting,,1503,91
Hitmonchan,Fighting,,1527,91
Lickitung,Normal,,1638,154
Koffing,Poison,,1160,75
Weezing,Poison,,2266,115
Rhyhorn,Ground,Rock,1190,138
Rhydon,Ground,Rock,2259,178
Chansey,Normal,,679,408
Tangela,Grass,,1752,115
Kangaskhan,Normal,,2057,178
Horsea,Water,,800,59
Seadra,Water,,1725,99
Goldeen,Water,,972,83
Seaking,Water,,2058,138
Staryu,Water,,944,59
Starmie,Water,Psychic,2197,107
Mr. Mime,Psychic,Fairy,1505,75
Scyther,Bug,Flying,2088,122
Jynx,Ice,Psychic,1728,115
Electabuzz,Electric,,2134,115
Magmar,Fire,,2281,115
Pinsir,Bug,,2137,115
Tauros,Normal,,1857,130
Magikarp,Water,,264,43
Gyarados,Water,Flying,2708,162
Lapras,Water,Ice,3002,218
Ditto,Normal,,926,88
Eevee,Normal,,1084,99
Vaporeon,Water,,2836,218
Jolteon,Electric,,2155,115
Flareon,Fire,,2662,115
Porygon,Normal,,1703,115
Omanyte,Rock,Water,1127,67
Omastar,Rock,Water,2249,122
Kabuto,Rock,Water,1112,59
Kabutops,Rock,Water,2145,107
Aerodactyl,Rock,Flying,2180,138
Snorlax,Normal,,3135,265
Articuno,Ice,Flying,2999,154
Zapdos,Electric,Flying,3136,154
Moltres,Fire,Flying,3263,154
Dratini,Dragon,,990,76
Dragonair,Dragon,,1760,108
Dragonite,Dragon,Flying,3525,156
Mewtwo,Psychic,,4174,180
Mew,Psychic,,3322,170

# model.py

from aredis_om import HashModel,JsonModel, NotFoundError , Field 
from pydantic import BaseModel


class Definitions(JsonModel, BaseModel):
    Name: str = Field(index=True, full_text_search=True)
    Type_1: str = Field(index=True, full_text_search=True)
    Type_2: str = Field(index=True, full_text_search=True)
    Max_CP:int
    Max_HP:int

# data upload file

from csv import DictReader

import aioredis
import asyncio
from aredis_om import HashModel, NotFoundError  
from aredis_om import get_redis_connection
from aredis_om import Migrator
from model import Definitions

async def main():
    with open("pokemonGO.csv", 'r') as f:

        dict_reader = DictReader(f)

        list_of_dict = list(dict_reader)
        for i in list_of_dict:
            print(i)
            d = Definitions(**i)
            await d.save()

    
    await Migrator().run()

loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()

# fastapi file

import asyncio

from fastapi import FastAPI, HTTPException
from starlette.requests import Request
from starlette.responses import Response
from aredis_om import HashModel, NotFoundError , Migrator 
from aredis_om import get_redis_connection


from model import Definitions


from redis import ResponseError
from redisearch import Client, IndexDefinition, TextField, NumericField, TagField, GeoField
from redisearch import reducers
from redisearch.aggregation import AggregateRequest, Asc, Desc

def initializeClient():
    SCHEMA = (
        TextField("Name"),
        TextField("Type_1"),
        TextField("Type_2"),
        NumericField("Max_CP"),
        NumericField("Max_HP")
    )
    client =  Client("pokedex", host=<host>, port=12830, conn=None, password=<password>, decode_responses=True)
    definition = IndexDefinition(prefix=[':model.Definitions:'])
    try:
        client.info()
    except ResponseError:
        # Index does not exist. We need to create it!
        client.create_index(SCHEMA, definition=definition)
    return client


client = initializeClient()

app = FastAPI()

@app.post("/pokemon")
async def save_pkemon(pokemon: Definitions):
    return await pokemon.save() 

@app.get("/search/{keyword}")
async def search_pokemon(keyword:str):
    return client.search(f"@Name:{keyword}") 

@app.get("/search-type1/{keyword}")
async def search_pokemon(keyword:str):
    return client.search(f"@Type_1:{keyword}*") 

@app.get("/search-type2/{keyword}")
async def search_pokemon(keyword:str):
    return client.search(f"@Type_2:{keyword}*")

@app.get("/fuzyy-search/{keyword}")
async def fuzzy_pokemons(keyword:str):
    return await Definitions.find(Definitions.Name % keyword).all()

@app.get("/pokemon/")
async def list_pokemons():
    return await Definitions.find().all()