# redis
> **Content description**: How to create a cluster of Redis to perform a fast and scalable search in python.
---
Hi readers, after reading this blog I hope you will be having a feeling like…
You can also choose to read the basics or dive right away into coding if you are here just for code reference. way easy than choosing a red pill and a blue pill right…
**So let's address the elephant in the room, what is Redis?**
*Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes with radius queries and streams. — Redis Website*
Wow ! That is too technical right ? It is just a type of db where data is stored in memory (RAM) instead of disk with additional out of the box features like hashes, object mapping and what not !!.
Now as you already have fair idea of what is redis lets dive into redis cluster. Wait what !!!, yes you read that right, as you know what is redis just add more redis and more redis and more of it and make all them talk in different machines in different countries, That is what redis cluster is all about. Check yourselves with below image.
Now Lets dive into some technical as this is a tech blog..
* Every Redis Cluster node requires two TCP connections. The normal Redis TCP port used to serve clients, i.e. 6379, plus the port obtained by adding 10000 to the data port, so 16379. Make sure you open both ports in your firewall, otherwise Redis cluster nodes will be not able to communicate.
* Redis Cluster does not use consistent hashing, but a different form of sharding where every key is conceptually part of what we call an hash slot. There are 16384 hash slots in Redis Cluster.
* Every node in a Redis Cluster is responsible for a subset of the hash slots, so for example you may have a cluster with 3 nodes, where:
* Node A contains hash slots from 0 to 5500.
* Node B contains hash slots from 5501 to 11000.
* Node C contains hash slots from 11001 to 16383.
Now we know what it is how can we get our hands dirty.
There are 3 ways to setup a cluster for testing and experimenting we are not going to explain everything still we add useful links which you can refer and get started.
## WAY 1
#### Extermely Easy: just a few clicks (Go Enterprise)
* [Try Redis Cloud for free](https://redis.info/3NBGJRT)
* They setup everything with autoscalling, clusters with fail proof systems, and so much more.
* [Watch this video on the benefits of Redis Cloud over other Redis providers](https://redis.info/3Ga9YII)
* This will show you the ultimate flexibility and capabilities of redis & enterprise.
* [Redis Developer Hub - tools, guides, and tutorials about Redis](https://redis.info/3LC4GqB)
* You'll find the most important and great tutorials to get you started witht the basics and advanced stuff with redis. They also provide the datasets which can be used for testing out the capacitites and operations for your side projects along with redis. :wink:
* [RedisInsight Desktop GUI](https://redis.info/3wMR7PR)
* This will help you visualise all the data and manage multiple redis databases in a cool GUI.
## WAY 2
#### Tedious way: Binary installs and running:
Well not that tedious, and later you can build a bash scrip to start the spinning up process.
#### Steps:
- Creating folders:
- As We'll be building a master-slave cluster aka. HA Cluster (i.e. there will be a pair of redis instances on a particular node for a cluster)
- Lets start by creatign multiple folders viz. The number of nodes * 2: 3 * 2 = 6 folders (planning to have 3 masters and 3 slaves)
- You can name these folders as simple as the port numbers they'll be listening to (just a suggestion :wink:) or any other significant nomenclature your heart desires.
- In each of these folders you'll need to download, unzip and make redis. Follow the below commands for every folder.
```
$ wget http://download.redis.io/releases/redis-7.0.4.tar.gz
$ tar xzf redis-7.0.4.tar.gz
$ cd redis-7.0.4
$ make
```
**Pro tip**: You can also simply download the package in a folder and copy it to other folders. This will save you some download time and bandwidth.
- Spinning up Redis in cluster mode.
- create/edit "redis.conf" file in each folder to have these contents to configure redis for cluster mode.
```
port <Port number>
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
```
- to eneable cluster-mode its as easy as enabling "cluster-enabled" directive.
- Now go ahead, start 6 terminals and navigate to all these folders, and start redis with command:
```
./src/redis-server ./redis.conf
```
- Open a 7th terminal to create the cluster:
```
./redis-cli --cluster create 127.0.0.1:<port for 1 redis> \
127.0.0.1:<port for 2 redis> 127.0.0.1:<port for 3 redis> \
127.0.0.1:<port for 4 redis> 127.0.0.1:<port for 5 redis> \
127.0.0.1:<port for 6 redis>
--cluster-replicas 1
```
- "--cluster-replicas" flag set to 1 means that every master has one slave when created.
- After this you'll see a lot of logs for "performing check", "Adding replica" and "eassigning each epoch"
- Connect with the cluster using redis-cli by running
``` ./src/redis-cli -c -p <port>```
You can try a SET and GET operations to test the cluster.
- Adding RediSearch and Redis Insight.
- The best part of redis modules is that they sit on top of cluster nodes and provide all this awesome support with a blazingly fast response. **I beg to differ that redis could be the only database needed for your BE**.
- RediSearch, RedisJSON, RedisTImeSeries, RedisGraph, etc. Covers everything a BE needs.
- And maintaining a structure for BE became easier with RedisOM which is a ODM for redis.
- Download :arrow_down: redis enterprise https://redis.com/redis-enterprise-software/download-center/software/ as well as Redis insight https://redis.com/redis-enterprise/redis-insight/
- after downloading open Redis insight:
- Register your cluster by clicking on the "+Add Redis DataBase"
- Do the defaults by adding the Alias name, url and port.
- Add redis search as a module by following the guide: https://docs.redis.com/latest/modules/install/add-module-to-cluster/
I simply did:
```
POST https://[host][:port]/v1/modules
{"module=@/tmp/redisearch.Linux-ubuntu16.04-x86_64.2.2.6.zip"}
```
## WAY 3
#### Slightly easier than going all the way down with manual configs.
- Use docker-compose to host a cluster.
- Use Redis Enterprise & Insight to setup RediSearch.
- And voila!!!:confetti_ball:
- Use this for reference : https://pierreabreu.medium.com/building-redis-cluster-with-docker-compose-9569ddb6414a
> Play around with Redis Insight with the simple operatios and explore the stack section for tutorials there.
## Lets Implement a BE to see redis in action
I wish to explore on the search features of Redis combined with the Hash/Json module for redis.
#### We will create a small microservice for extending and interacting with redis.
### env setup for redis OM
```
$ export export REDIS_OM_URL=redis://<username>:<password>@<host>:<port>
```
### For fun lets use some pokemon data:
We will upload this data through the redisOM ODM later.
Create a file called pokemon.csv
```
Name,Type_1,Type_2,Max_CP,Max_HP
Bulbasaur,Grass,Poison,1079,83
Ivysaur,Grass,Poison,1643,107
Venusaur,Grass,Poison,2598,138
Charmander,Fire,,962,73
Charmeleon,Fire,,1568,103
Charizard,Fire,Flying,2620,135
Squirtle,Water,,1015,81
Wartortle,Water,,1594,105
Blastoise,Water,,2560,137
Caterpie,Bug,,446,83
Metapod,Bug,,481,91
Butterfree,Bug,Flying,1465,107
Weedle,Bug,Poison,452,75
Kakuna,Bug,Poison,488,83
Beedrill,Bug,Poison,1450,115
Pidgey,Normal,Flying,684,75
Pidgeotto,Normal,Flying,1232,111
Pidgeot,Normal,Flying,2106,143
Rattata,Normal,,585,59
Raticate,Normal,,1454,99
Spearow,Normal,Flying,691,75
Fearow,Normal,Flying,1758,115
Ekans,Poison,,830,67
Arbok,Poison,,1779,107
Pikachu,Electric,,894,67
Raichu,Electric,,2042,107
Sandshrew,Ground,,804,91
Sandslash,Ground,,1823,130
Nidoran♀,Poison,,882,99
Nidorina,Poison,,1414,122
Nidoqueen,Poison,Ground,2502,154
Nidoran♂,Poison,,849,84
Nidorino,Poison,,1382,108
Nidoking,Poison,Ground,2492,140
Clefairy,Fairy,,1209,122
Clefable,Fairy,,2414,162
Vulpix,Fire,,837,72
Ninetales,Fire,,2203,127
Jigglypuff,Normal,Fairy,924,194
Wigglytuff,Normal,Fairy,2192,233
Zubat,Poison,Flying,647,75
Golbat,Poison,Flying,1935,130
Oddish,Grass,Poison,1156,83
Gloom,Grass,Poison,1701,107
Vileplume,Grass,Poison,2510,130
Paras,Bug,Grass,923,67
Parasect,Bug,Grass,1759,107
Venonat,Bug,Poison,1036,107
Venomoth,Bug,Poison,1903,122
Diglett,Ground,,460,27
Dugtrio,Ground,,1176,67
Meowth,Normal,,761,75
Persian,Normal,,1643,115
Psyduck,Water,,1117,91
Golduck,Water,,2403,138
Mankey,Fighting,,884,75
Primeape,Fighting,,1877,115
Growlithe,Fire,,1344,99
Arcanine,Fire,,3005,154
Poliwag,Water,,801,75
Poliwhirl,Water,,1350,115
Poliwrath,Water,Fighting,2523,154
Abra,Psychic,,604,51
Kadabra,Psychic,,1140,75
Alakazam,Psychic,,1826,99
Machop,Fighting,,1097,122
Machoke,Fighting,,1773,138
Machamp,Fighting,,2612,154
Bellsprout,Grass,Poison,1125,91
Weepinbell,Grass,Poison,1736,115
Victreebel,Grass,Poison,2548,138
Tentacool,Water,Poison,911,75
Tentacruel,Water,Poison,2236,138
Geodude,Rock,Ground,855,75
Graveler,Rock,Ground,1443,99
Golem,Rock,Ground,2319,138
Ponyta,Fire,,1526,91
Rapidash,Fire,,2215,115
Slowpoke,Water,Psychic,1227,154
Slowbro,Water,Psychic,2615,162
Magnemite,Electric,Steel,897,51
Magneton,Electric,Steel,1893,91
Farfetch'd,Normal,Flying,1272,94
Doduo,Normal,Flying,861,67
Dodrio,Normal,Flying,1849,107
Seel,Water,,1114,115
Dewgong,Water,Ice,2161,154
Grimer,Poison,,1293,138
Muk,Poison,,2621,178
Shellder,Water,,828,59
Cloyster,Water,Ice,2067,91
Gastly,Ghost,Poison,810,59
Haunter,Ghost,Poison,1390,83
Gengar,Ghost,Poison,2093,107
Onix,Rock,Ground,863,67
Drowzee,Psychic,,1082,107
Hypno,Psychic,,2199,146
Krabby,Water,,797,59
Kingler,Water,,1836,99
Voltorb,Electric,,845,75
Electrode,Electric,,1657,107
Exeggcute,Grass,Psychic,1107,107
Exeggutor,Grass,Psychic,2976,162
Cubone,Ground,,1013,91
Marowak,Ground,,1668,107
Hitmonlee,Fighting,,1503,91
Hitmonchan,Fighting,,1527,91
Lickitung,Normal,,1638,154
Koffing,Poison,,1160,75
Weezing,Poison,,2266,115
Rhyhorn,Ground,Rock,1190,138
Rhydon,Ground,Rock,2259,178
Chansey,Normal,,679,408
Tangela,Grass,,1752,115
Kangaskhan,Normal,,2057,178
Horsea,Water,,800,59
Seadra,Water,,1725,99
Goldeen,Water,,972,83
Seaking,Water,,2058,138
Staryu,Water,,944,59
Starmie,Water,Psychic,2197,107
Mr. Mime,Psychic,Fairy,1505,75
Scyther,Bug,Flying,2088,122
Jynx,Ice,Psychic,1728,115
Electabuzz,Electric,,2134,115
Magmar,Fire,,2281,115
Pinsir,Bug,,2137,115
Tauros,Normal,,1857,130
Magikarp,Water,,264,43
Gyarados,Water,Flying,2708,162
Lapras,Water,Ice,3002,218
Ditto,Normal,,926,88
Eevee,Normal,,1084,99
Vaporeon,Water,,2836,218
Jolteon,Electric,,2155,115
Flareon,Fire,,2662,115
Porygon,Normal,,1703,115
Omanyte,Rock,Water,1127,67
Omastar,Rock,Water,2249,122
Kabuto,Rock,Water,1112,59
Kabutops,Rock,Water,2145,107
Aerodactyl,Rock,Flying,2180,138
Snorlax,Normal,,3135,265
Articuno,Ice,Flying,2999,154
Zapdos,Electric,Flying,3136,154
Moltres,Fire,Flying,3263,154
Dratini,Dragon,,990,76
Dragonair,Dragon,,1760,108
Dragonite,Dragon,Flying,3525,156
Mewtwo,Psychic,,4174,180
Mew,Psychic,,3322,170
```
## So what is the plan here ?
We will create a simple LookUp DB service for pokemon using redisOM python, redis search-py & fastapi
- FastApi for building a microservice
- redisOM as an ODM
- redisearch-py fro redissearch operations.
### lets create a file called model.py
Here we are going to create a DB Model analysing the data we have.
> Creating auto indexes over a field is as simple as setting index=True, as well as setting up full text search takes enabling full_text_search=True on a field
```
from aredis_om import HashModel,JsonModel, NotFoundError , Field
from pydantic import BaseModel
class Pokemons(HashModel, BaseModel):
Name: str = Field(index=True, full_text_search=True)
Type_1: str = Field(index=True, full_text_search=True)
Type_2: str = Field(index=True, full_text_search=True)
Max_CP:int
Max_HP:int
```
### Now lets upload the data to DB, create a file called uploader.py
```
from csv import DictReader
import aioredis
import asyncio
from aredis_om import HashModel, NotFoundError
from aredis_om import get_redis_connection
from aredis_om import Migrator
from model import Pokemons
async def main():
with open("pokemonGO.csv", 'r') as f:
dict_reader = DictReader(f)
list_of_dict = list(dict_reader)
for i in list_of_dict:
print(i)
d = Pokemons(**i)
await d.save()
await Migrator().run()
loop = asyncio.get_event_loop()
loop.run_until_complete(main())
loop.close()
```
## Now creatre a fastapi server file server.py
```
import asyncio
from fastapi import FastAPI, HTTPException
from starlette.requests import Request
from starlette.responses import Response
from aredis_om import HashModel, NotFoundError , Migrator
from aredis_om import get_redis_connectio
from model import Pokemons
from redis import ResponseError
from redisearch import Client, IndexDefinition, TextField, NumericField, TagField, GeoField
from redisearch import reducers
from redisearch.aggregation import AggregateRequest, Asc, Desc
```
- Redissearch-py initialising client.
```
def initializeClient():
SCHEMA = (
TextField("Name"),
TextField("Type_1"),
TextField("Type_2"),
NumericField("Max_CP"),
NumericField("Max_HP")
)
client = Client("pokedex", host=<host>, port=12830, conn=None, password=<password>, decode_responses=True)
definition = IndexDefinition(prefix=[':model.Pokemons:'])
try:
client.info()
except ResponseError:
# Index does not exist. We need to create it!
client.create_index(SCHEMA, definition=definition)
return client
client = initializeClient()
```
### Creating enpoints
Getting/Setting endpoints for our lovely pokemons.
#### Lets Go
This endpoint creates a pokemon, try sending it a raw JSON and you'll see the pokemon inserted in our DB.
```
app = FastAPI()
@app.post("/pokemon")
async def save_pkemon(pokemon: Pokemons):
return await pokemon.save()
```
- Performing search using redisearch-py client
The redis client can be used to search accross the DB for a specific attribute. Here we're creating endpoints for searching based on name, type 1 & 2 of the pokemon.
(In reality you'll be combining these endpoints for complete searching.)
```
@app.get("/search/{keyword}")
async def search_pokemon(keyword:str):
return client.search(f"@Name:{keyword}")
@app.get("/search-type1/{keyword}")
async def search_pokemon(keyword:str):
return client.search(f"@Type_1:{keyword}*")
@app.get("/search-type2/{keyword}")
async def search_pokemon(keyword:str):
return client.search(f"@Type_2:{keyword}*")
```
- Finding them using RedisOM
Lets do the same with the RedisOM which has a simpler syntax (according to me :wink:)
```
@app.get("/fuzyy-search/{keyword}")
async def fuzzy_pokemons(keyword:str):
return await Pokemons.find(Pokemons.Name % keyword).all()
```
- Listing all pokemons
Here Come the last endpoint where we'll just list all our **cutemon's**
```
@app.get("/pokemon/")
async def list_pokemons():
return await Pokemons.find().all()
```
- Finally, let's see this server performing for us.
This will initialize the server and generate our documentation for interacting with the endpoints (Swagger got the swag!!! :sunglasses:)
```
import uvicorn
if (__name__ == "__main__"):
uvicorn.run("main:app", port=8000, reload=True, access_log=False)
```
Run the main.py file
```
python main.py
```
In your browser visit http://localhost:8000/docs. You’ll be greeted with the Swagger/Open API docs with all the endpoints created above.
### Conclusion:
Conclusion:
The setup is easy if you know what's to be done and how things fit together, but then the responsibility of having to look up for scaling and failure goes up. Redis cloud is better suited for a developer who needs to move fast without worrying about infrastructure concerns. The ODM is easy to get going and can be used with JSON to model any complex data structure.
Redis in all is 🔥 with its capabilities!
Sign in with Wallet
Connect another wallet