Deploy a Pulsar cluster

tags: pulsar

Ensure you have Java 8 installed

sudo apt update && sudo apt -y install openjdk-8-jre-headless

Run a Zookeeper node if you haven’t do

curl https://archive.apache.org/dist/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz -o zookeeper-3.4.11.tar.gz
tar -xzf zookeeper-3.4.11.tar.gz
cp zookeeper-3.4.11/conf/zoo_sample.cfg zookeeper-3.4.11/conf/zoo.cfg
cd zookeeper-3.4.11
bin/zkServer.sh start conf/zoo.cfg

Download Apache Pulsar

wget https://archive.apache.org/dist/pulsar/pulsar-2.3.1/apache-pulsar-2.3.1-bin.tar.gz
tar xvzf apache-pulsar-2.3.1-bin.tar.gz
cd apache-pulsar-2.3.1

Ensure you have set env JAVA_HOME

export JAVA_HOME=/usr

Initialize cluster metadata

bin/pulsar initialize-cluster-metadata \
  --cluster pulsar-cluster-1 \
  --zookeeper 192.168.2.100:2181 \
  --configuration-store 192.168.2.100:2181 \
  --web-service-url http://192.168.2.100:8080 \
  --broker-service-url pulsar://192.168.2.100:6650

Provision a BookKeeper cluster

# Ensure configs below had been set correctly:
# advertisedAddress=
# zkServers=
# extraServerComponents=org.apache.bookkeeper.stream.server.StreamStorageLifecycleComponent
vim conf/bookkeeper.conf
# Start the bookie
bin/pulsar-daemon start bookie
# Test the BookKeeper cluster had been run
bin/bookkeeper shell bookiesanity
# (Run this if needed) Clear BAD state of BookKeeper states
bin/bookkeeper shell metaformat -f

Provision the Pulsar Brokers

• brokerServicePort: 6650
• webServicePort: 8080

# Ensure configs below had been set correctly:
# advertisedAddress=
# zookeeperServers=
# configurationStoreServers= <- set it eaqul to 'zookeeperServers' in most cases
# clusterName= <- the same as you assigned to stpe 'Initialize cluster metadata'
# managedLedgerDefaultEnsembleSize=1 <- Number of bookies to use when creating a ledger
# managedLedgerDefaultWriteQuorum=1 <- Number of copies to store for each message
# managedLedgerDefaultAckQuorum=1 <- Number of guaranteed copies (acks to wait before write is complete)
# functionsWorkerEnabled=true <- set to true if wants to enalbe Pulsar Functions
vim conf/broker.conf
# If wants to enable Pulsar Functions, ensure the cluster name is correct 
# pulsarFunctionsCluster: <- the same as you assigned to stpe 'Initialize cluster metadata'
vim conf/functions_worker.yml
# Start the Pulsar Broker
bin/pulsar-daemon start broker

Use pulsar-client to test the cluster had been provisioned correctly

# Ensure pulsar-client can connect to the Pulsar cluster
# webServiceUrl=
# brokerServiceurl=
vim conf/client.conf
# Start a consumer
bin/pulsar-client consume \
  persistent://public/default/test \
  -n 100 \
  -s "consumer-test" \
  -t "Exclusive"
# Produce a message
bin/pulsar-client produce \
  persistent://public/default/test \
  -n 1 \
  -m "Hello Pulsar"

Test Pulsar Functions

# Create a Pulsar Function
bin/pulsar-admin functions create \
  --jar examples/api-examples.jar \
  --classname org.apache.pulsar.functions.api.examples.ExclamationFunction \
  --inputs persistent://public/default/exclamation-input \
  --output persistent://public/default/exclamation-output \
  --tenant public \
  --namespace default \
  --name exclamation
# Trigger the Pulsar Function
bin/pulsar-admin functions trigger --name exclamation --trigger-value "hello world"

References

• Deploying a cluster on bare metal · Apache Pulsar