Ufdjrw

I am reading 500 MB random tuples from Kafka producer continuous and in a storm topology I am inserting it to MongoDb using Mongo Java Driver. The problem is I am getting really low throughput as 4-5 tuples per second.

Without DB insert if I write a simple print statement I get throughput as 684 tuples per second. I am planning to run 1Million records from Kafka and check the throughput with mongo insert.

I tried to tune using config setMaxSpoutPending , setMessageTimeoutSecs parms in kafkaconfig.

 final SpoutConfig kafkaConf = new SpoutConfig(zkrHosts, kafkaTopic, zkRoot, clientId);
 kafkaConf.ignoreZkOffsets=false;
 kafkaConf.useStartOffsetTimeIfOffsetOutOfRange=true;
 kafkaConf.startOffsetTime=kafka.api.OffsetRequest.LatestTime();
 kafkaConf.stateUpdateIntervalMs=2000;
 kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
 final TopologyBuilder topologyBuilder = new TopologyBuilder();
 topologyBuilder.setSpout("kafka-spout", new KafkaSpout(kafkaConf), 1);
 topologyBuilder.setBolt("print-messages", new MyKafkaBolt()).shuffleGrouping("kafka-spout");
 Config conf = new Config();
 conf.setDebug(true);
 conf.setMaxSpoutPending(1000);
 conf.setMessageTimeoutSecs(30);

Execute method of bolt

 JSONObject jObj = new JSONObject();
 jObj.put("key", input.getString(0));

 if (null !=jObj && jObj.size() > 0 ) {
 final DBCollection quoteCollection = dbConnect.getConnection().getCollection("stormPoc");
 if (quoteCollection != null) 
 BasicDBObject dbObject = new BasicDBObject();
 dbObject.putAll(jObj);
 quoteCollection.insert(dbObject);
 // logger.info("inserted in Collection !!!");
 else 
 logger.info("Error while inserting data in DB!!!");
 
 collector.ack(input);

There is a storm-mongodb module for integration with Mongo. Does it not do the job? https://github.com/apache/storm/tree/b07413670fa62fec077c92cb78fc711c3bda820c/external/storm-mongodb

You shouldn't use storm-kafka for Kafka integration, it is deprecated. Use storm-kafka-client instead.

Setting conf.setDebug(true) will impact your processing, as Storm will log a fairly huge amount of text per tuple.