Logstash {:exception=>#Unable to insert json arrays from Logstash to mongoDBLogstash/Graylog won't write to ES with data node downLogstash 5.1.1 “bad URI(is not URI?)"When I ./logstash -f test.conf, it shows the errorLogstash stopping randomly after few hoursAdding logstash “logstash-output-email” failed and restarting logstash failed to startlogstash is throwing exception template file not foundLogstash got exception in runningHi All, We are trying to push the data from csv file into elastic search but we are facing the below issue?

Why is delta-v is the most useful quantity for planning space travel?

How do ultrasonic sensors differentiate between transmitted and received signals?

How can I successfully establish a nationwide combat training program for a large country?

When is separating the total wavefunction into a space part and a spin part possible?

What was required to accept "troll"?

How can I raise concerns with a new DM about XP splitting?

Superhero words!

Is it possible to build a CPA Secure encryption scheme which remains secure even when the encryption of secret key is given?

Why are on-board computers allowed to change controls without notifying the pilots?

Can a controlled ghast be a leader of a pack of ghouls?

Are taller landing gear bad for aircraft, particulary large airliners?

Latex for-and in equation

Bob has never been a M before

Is there enough fresh water in the world to eradicate the drinking water crisis?

Invariance of results when scaling explanatory variables in logistic regression, is there a proof?

Pronouncing Homer as in modern Greek

Freedom of speech and where it applies

Is the next prime number always the next number divisible by the current prime number, except for any numbers previously divisible by primes?

A car is moving at 40 km/h. A fly at 100 km/h, starts from wall towards the car(20 km away)flies to car and back. How many trips can it make?

Resetting two CD4017 counters simultaneously, only one resets

Simple recursive Sudoku solver

Organic chemistry Iodoform Reaction

What (else) happened July 1st 1858 in London?

Can I rely on these GitHub repository files?



Logstash {:exception=>#


Unable to insert json arrays from Logstash to mongoDBLogstash/Graylog won't write to ES with data node downLogstash 5.1.1 “bad URI(is not URI?)"When I ./logstash -f test.conf, it shows the errorLogstash stopping randomly after few hoursAdding logstash “logstash-output-email” failed and restarting logstash failed to startlogstash is throwing exception template file not foundLogstash got exception in runningHi All, We are trying to push the data from csv file into elastic search but we are facing the below issue?













-2















I am using Logstash running in docker to update indices on amazon elastic search using input from JDBC plugin. It picks up data from mysql database on Amazon RDS and updating the indices.
I have pre-configured docker build for this and using the same for running a logstash service on AWS ECS. I am new to entire ELK stack so I am not aware if I am missing any configuration setting but the same configurations have been in use for quite some time and worked fine till now.



Now I have changes some SQL queries in jdbc input plugin , in the configuration file and running the docker on local with VPN to connect to AWS RDS .
On running logstash docker URI::InvalidComponentError: bad component(expected port component): 443 error shows. Below is full error :




09:41:44.412 [[main]-pipeline-manager] ERROR logstash.pipeline - Error registering plugin :plugin=>"#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3"]>, @metric=#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#>, @out_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: out value: 0, @in_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: in value: 0, @strategy=#[//search-raxa-es-5p5-vjyeulbwj2ybskndxeugwi2hsq.ap-southeast-1.es.amazonaws.com], region=>"ap-southeast-1", aws_access_key_id=>"AKIAJ5P6OIPWK3RK4L5A", aws_secret_access_key=>"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", index=>"public-investigations-signed-start-2019.02.25", document_id=>"logstash_investigation_SHA1_%conceptId_%fingerprint", id=>"46d8f17835109d9c9cc85703789ad3df34e27b10-3", enable_metric=>true, codec=>"plain_772781e2-1dc8-49e8-8077-bf704a8d47f9", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", port=>443, protocol=>"https", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, metric=>#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3"]>>>, @id="46d8f17835109d9c9cc85703789ad3df34e27b10-3", @time_metric=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: duration_in_millis value: 0, @metric_events=#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events]>, @output_class=LogStash::Outputs::ElasticSearch>", :error=>"bad component(expected port component): 443"
09:41:44.420 [[main]-pipeline-manager] ERROR logstash.agent - Pipeline aborted due to error :exception=>#<URI::InvalidComponentError: bad component(expected port component): 443>, :backtrace=>["/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:697:in check_port'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:744:inport='", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:195:in initialize'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/http.rb:84:ininitialize'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:140:in build'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/http.rb:62:inbuild'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:96:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:245:in healthcheck!'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:241:in healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:341:inupdate_urls'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:71:in start'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:316:inbuild_pool'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:66:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:108:increate_http_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:98:in build'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es.rb:253:inbuild_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:25:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:408:instart_pipeline'"]




The amazon_es output config is :



output {
 if "investigations" in [tags] 
 amazon_es 
 hosts => ["$AWS_ES_ENDPOINT"]
 region => "$AWS_ES_REGION"
 # aws_access_key_id, aws_secret_access_key optional if instance 
 profile is configured
 aws_access_key_id => '$LOGSTASH_WRITE_ES_ACCESS_KEY'
 aws_secret_access_key => '$LOGSTASH_WRITE_ES_SECRET_KEY'
 index => "my_index_name"
 document_id => "logstash_%documentId_%fingerprint"


I tried adding the port to the host and port component in config nothing seem to work and a few other solutions . Please comment if any other part of the configuration is needed, I will update the same.






ruby amazon-web-services logstash dockerfile logstash-configuration







share|improve this question




























    -2















    I am using Logstash running in docker to update indices on amazon elastic search using input from JDBC plugin. It picks up data from mysql database on Amazon RDS and updating the indices.
    I have pre-configured docker build for this and using the same for running a logstash service on AWS ECS. I am new to entire ELK stack so I am not aware if I am missing any configuration setting but the same configurations have been in use for quite some time and worked fine till now.



    Now I have changes some SQL queries in jdbc input plugin , in the configuration file and running the docker on local with VPN to connect to AWS RDS .
    On running logstash docker URI::InvalidComponentError: bad component(expected port component): 443 error shows. Below is full error :




    09:41:44.412 [[main]-pipeline-manager] ERROR logstash.pipeline - Error registering plugin :plugin=>"#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3"]>, @metric=#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#>, @out_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: out value: 0, @in_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: in value: 0, @strategy=#[//search-raxa-es-5p5-vjyeulbwj2ybskndxeugwi2hsq.ap-southeast-1.es.amazonaws.com], region=>"ap-southeast-1", aws_access_key_id=>"AKIAJ5P6OIPWK3RK4L5A", aws_secret_access_key=>"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", index=>"public-investigations-signed-start-2019.02.25", document_id=>"logstash_investigation_SHA1_%conceptId_%fingerprint", id=>"46d8f17835109d9c9cc85703789ad3df34e27b10-3", enable_metric=>true, codec=>"plain_772781e2-1dc8-49e8-8077-bf704a8d47f9", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", port=>443, protocol=>"https", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, metric=>#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3"]>>>, @id="46d8f17835109d9c9cc85703789ad3df34e27b10-3", @time_metric=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: duration_in_millis value: 0, @metric_events=#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events]>, @output_class=LogStash::Outputs::ElasticSearch>", :error=>"bad component(expected port component): 443"
    09:41:44.420 [[main]-pipeline-manager] ERROR logstash.agent - Pipeline aborted due to error :exception=>#<URI::InvalidComponentError: bad component(expected port component): 443>, :backtrace=>["/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:697:in check_port'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:744:inport='", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:195:in initialize'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/http.rb:84:ininitialize'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:140:in build'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/http.rb:62:inbuild'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:96:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:245:in healthcheck!'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:241:in healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:341:inupdate_urls'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:71:in start'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:316:inbuild_pool'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:66:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:108:increate_http_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:98:in build'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es.rb:253:inbuild_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:25:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:408:instart_pipeline'"]




    The amazon_es output config is :



    output {
     if "investigations" in [tags] 
     amazon_es 
     hosts => ["$AWS_ES_ENDPOINT"]
     region => "$AWS_ES_REGION"
     # aws_access_key_id, aws_secret_access_key optional if instance 
     profile is configured
     aws_access_key_id => '$LOGSTASH_WRITE_ES_ACCESS_KEY'
     aws_secret_access_key => '$LOGSTASH_WRITE_ES_SECRET_KEY'
     index => "my_index_name"
     document_id => "logstash_%documentId_%fingerprint"


    I tried adding the port to the host and port component in config nothing seem to work and a few other solutions . Please comment if any other part of the configuration is needed, I will update the same.






    ruby amazon-web-services logstash dockerfile logstash-configuration







    share|improve this question


























      -2












      -2








      -2








      I am using Logstash running in docker to update indices on amazon elastic search using input from JDBC plugin. It picks up data from mysql database on Amazon RDS and updating the indices.
      I have pre-configured docker build for this and using the same for running a logstash service on AWS ECS. I am new to entire ELK stack so I am not aware if I am missing any configuration setting but the same configurations have been in use for quite some time and worked fine till now.



      Now I have changes some SQL queries in jdbc input plugin , in the configuration file and running the docker on local with VPN to connect to AWS RDS .
      On running logstash docker URI::InvalidComponentError: bad component(expected port component): 443 error shows. Below is full error :




      09:41:44.412 [[main]-pipeline-manager] ERROR logstash.pipeline - Error registering plugin :plugin=>"#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3"]>, @metric=#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#>, @out_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: out value: 0, @in_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: in value: 0, @strategy=#[//search-raxa-es-5p5-vjyeulbwj2ybskndxeugwi2hsq.ap-southeast-1.es.amazonaws.com], region=>"ap-southeast-1", aws_access_key_id=>"AKIAJ5P6OIPWK3RK4L5A", aws_secret_access_key=>"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", index=>"public-investigations-signed-start-2019.02.25", document_id=>"logstash_investigation_SHA1_%conceptId_%fingerprint", id=>"46d8f17835109d9c9cc85703789ad3df34e27b10-3", enable_metric=>true, codec=>"plain_772781e2-1dc8-49e8-8077-bf704a8d47f9", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", port=>443, protocol=>"https", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, metric=>#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3"]>>>, @id="46d8f17835109d9c9cc85703789ad3df34e27b10-3", @time_metric=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: duration_in_millis value: 0, @metric_events=#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events]>, @output_class=LogStash::Outputs::ElasticSearch>", :error=>"bad component(expected port component): 443"
      09:41:44.420 [[main]-pipeline-manager] ERROR logstash.agent - Pipeline aborted due to error :exception=>#<URI::InvalidComponentError: bad component(expected port component): 443>, :backtrace=>["/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:697:in check_port'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:744:inport='", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:195:in initialize'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/http.rb:84:ininitialize'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:140:in build'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/http.rb:62:inbuild'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:96:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:245:in healthcheck!'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:241:in healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:341:inupdate_urls'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:71:in start'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:316:inbuild_pool'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:66:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:108:increate_http_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:98:in build'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es.rb:253:inbuild_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:25:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:408:instart_pipeline'"]




      The amazon_es output config is :



      output {
       if "investigations" in [tags] 
       amazon_es 
       hosts => ["$AWS_ES_ENDPOINT"]
       region => "$AWS_ES_REGION"
       # aws_access_key_id, aws_secret_access_key optional if instance 
       profile is configured
       aws_access_key_id => '$LOGSTASH_WRITE_ES_ACCESS_KEY'
       aws_secret_access_key => '$LOGSTASH_WRITE_ES_SECRET_KEY'
       index => "my_index_name"
       document_id => "logstash_%documentId_%fingerprint"


      I tried adding the port to the host and port component in config nothing seem to work and a few other solutions . Please comment if any other part of the configuration is needed, I will update the same.






      ruby amazon-web-services logstash dockerfile logstash-configuration







      share|improve this question
















      I am using Logstash running in docker to update indices on amazon elastic search using input from JDBC plugin. It picks up data from mysql database on Amazon RDS and updating the indices.
      I have pre-configured docker build for this and using the same for running a logstash service on AWS ECS. I am new to entire ELK stack so I am not aware if I am missing any configuration setting but the same configurations have been in use for quite some time and worked fine till now.



      Now I have changes some SQL queries in jdbc input plugin , in the configuration file and running the docker on local with VPN to connect to AWS RDS .
      On running logstash docker URI::InvalidComponentError: bad component(expected port component): 443 error shows. Below is full error :




      09:41:44.412 [[main]-pipeline-manager] ERROR logstash.pipeline - Error registering plugin :plugin=>"#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3"]>, @metric=#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#>, @out_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: out value: 0, @in_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: in value: 0, @strategy=#[//search-raxa-es-5p5-vjyeulbwj2ybskndxeugwi2hsq.ap-southeast-1.es.amazonaws.com], region=>"ap-southeast-1", aws_access_key_id=>"AKIAJ5P6OIPWK3RK4L5A", aws_secret_access_key=>"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", index=>"public-investigations-signed-start-2019.02.25", document_id=>"logstash_investigation_SHA1_%conceptId_%fingerprint", id=>"46d8f17835109d9c9cc85703789ad3df34e27b10-3", enable_metric=>true, codec=>"plain_772781e2-1dc8-49e8-8077-bf704a8d47f9", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", port=>443, protocol=>"https", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false, metric=>#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3"]>>>, @id="46d8f17835109d9c9cc85703789ad3df34e27b10-3", @time_metric=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events] key: duration_in_millis value: 0, @metric_events=#, @structured_lookup_mutex=#, @fast_lookup=#>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"46d8f17835109d9c9cc85703789ad3df34e27b10-3", :events]>, @output_class=LogStash::Outputs::ElasticSearch>", :error=>"bad component(expected port component): 443"
      09:41:44.420 [[main]-pipeline-manager] ERROR logstash.agent - Pipeline aborted due to error :exception=>#<URI::InvalidComponentError: bad component(expected port component): 443>, :backtrace=>["/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:697:in check_port'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:744:inport='", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:195:in initialize'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/http.rb:84:ininitialize'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/generic.rb:140:in build'", "/usr/share/logstash/vendor/jruby/lib/ruby/1.9/uri/http.rb:62:inbuild'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:96:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:inperform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:245:in healthcheck!'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:241:in healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:341:inupdate_urls'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:71:in start'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:316:inbuild_pool'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:66:in initialize'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:108:increate_http_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:98:in build'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es.rb:253:inbuild_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:25:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "org/jruby/RubyArray.java:1613:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:408:instart_pipeline'"]




      The amazon_es output config is :



      output {
       if "investigations" in [tags] 
       amazon_es 
       hosts => ["$AWS_ES_ENDPOINT"]
       region => "$AWS_ES_REGION"
       # aws_access_key_id, aws_secret_access_key optional if instance 
       profile is configured
       aws_access_key_id => '$LOGSTASH_WRITE_ES_ACCESS_KEY'
       aws_secret_access_key => '$LOGSTASH_WRITE_ES_SECRET_KEY'
       index => "my_index_name"
       document_id => "logstash_%documentId_%fingerprint"


      I tried adding the port to the host and port component in config nothing seem to work and a few other solutions . Please comment if any other part of the configuration is needed, I will update the same.






      ruby amazon-web-services logstash dockerfile logstash-configuration




      ruby amazon-web-services logstash dockerfile logstash-configuration






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited Mar 8 at 6:12







      Anurag Shukla

















      asked Mar 7 at 9:58









      Anurag ShuklaAnurag Shukla

      158110




      158110






















          0






          active

          oldest

          votes











          Your Answer






          StackExchange.ifUsing("editor", function ()
          StackExchange.using("externalEditor", function ()
          StackExchange.using("snippets", function ()
          StackExchange.snippets.init();
          );
          );
          , "code-snippets");

          StackExchange.ready(function()
          var channelOptions =
          tags: "".split(" "),
          id: "1"
          ;
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function()
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled)
          StackExchange.using("snippets", function()
          createEditor();
          );

          else
          createEditor();

          );

          function createEditor()
          StackExchange.prepareEditor(
          heartbeatType: 'answer',
          autoActivateHeartbeat: false,
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader:
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          ,
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          );



          );













          draft saved

          draft discarded


















          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55040876%2flogstash-exception-uriinvalidcomponenterror-bad-componentexpected-port%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown

























          0






          active

          oldest

          votes








          0






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes















          draft saved

          draft discarded
















































          Thanks for contributing an answer to Stack Overflow!


          • Please be sure to answer the question. Provide details and share your research!

          But avoid …


          • Asking for help, clarification, or responding to other answers.

          • Making statements based on opinion; back them up with references or personal experience.

          To learn more, see our tips on writing great answers.




          draft saved


          draft discarded














          StackExchange.ready(
          function ()
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55040876%2flogstash-exception-uriinvalidcomponenterror-bad-componentexpected-port%23new-answer', 'question_page');

          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          -

          Popular posts from this blog

          AWS Lex not identifying response if by a variable The 2019 Stack Overflow Developer Survey Results Are In Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 17/18, 2019 at 00:00UTC (8:00pm US/Eastern) The Ask Question Wizard is Live! Data science time! April 2019 and salary with experienceEnforcing custom enumeration in AWS LEX for slot valuesHow to give response based on user response in Amazon Lex?Intercepting AWS Lambda Response to a AWS Lex QueryLex chat bot error: Reached second execution of fulfillment lambda on the same utteranceamazon lex showing invalid responseLambda response send back to Lex slot?Response card in Amazon lexAmazon Lex - Lambda response return HTML to botHow can I solve 424 (Failed Dependency) (python) obtained from Amazon lex?

          Image
          Do working physicists consider Newtonian mechanics to be "falsified"? Match Roman Numerals Derivation tree not rendering How can I protect witches in combat who wear limited clothing? Is above average number of years spent on PhD considered a red flag in future academia or industry positions? Relations between two reciprocal partial derivatives? Take groceries in checked luggage He got a vote 80% that of Emmanuel Macron’s Why not take a picture of a closer black hole? Is every episode of "Where are my Pants?" identical? Problems with Ubuntu mount /tmp Why is superheterodyning better than direct conversion? Are spiders unable to hurt humans, especially very small spiders? Can undead you have reanimated wait inside a portable hole? What is special about square numbers here? Python - Fishing Simulator Is this wall load bearing? Blueprints and photos attached Can a 1st-level character have an ability score above 18? Semisimplicity of ...
          Read more

          Алба-Юлія

          Image
          Алба-Юлія рум.  Alba Iulia герб прапор Православний собор в Алба-Юлії Основні дані 46°04′35″ пн. ш. 23°34′22″ сх. д.  /  46.07638888891677453° пн. ш. 23.57277777780577921° сх. д.  / 46.07638888891677453; 23.57277777780577921 Координати: 46°04′35″ пн. ш. 23°34′22″ сх. д.  /  46.07638888891677453...
          Read more

          Захаров Федір Захарович

          Image
          Федір Захарович Захаров Федір Захаров, 1986 рік Народження 5 вересня 1919 ( 1919-09-05 ) Олександрівське [d] , Смоленська губернія , Російська СФРР Смерть 29 вересня 1997 ( 1997-09-29 ) (78 років)   Ялта , Автономн...
          Read more