Ufdjrw

Has the laser at Magurele, Romania reached a tenth of the Sun's power?

Giving feedback to someone without sounding prejudiced

How do you justify more code being written by following clean code practices?

Is there a distance limit for minecart tracks?

If the only attacker is removed from combat, is a creature still counted as having attacked this turn?

Ways of geometrical multiplication

Would this string work as string?

What does "Scientists rise up against statistical significance" mean? (Comment in Nature)

Why didn’t Eve recognize the little cockroach as a living organism?

How can I, as DM, avoid the Conga Line of Death occurring when implementing some form of flanking rule?

What's the name of the logical fallacy where a debater extends a statement far beyond the original statement to make it true?

How to make money from a browser who sees 5 seconds into the future of any web page?

What is the meaning of "You've never met a graph you didn't like?"

Can I run 125kHz RF circuit on a breadboard?

Does the Crossbow Expert feat's extra crossbow attack work with the reaction attack from a Hunter ranger's Giant Killer feature?

"Oh no!" in Latin

Why is participating in the European Parliamentary elections used as a threat?

Personal or impersonal in a technical resume

How do I fix the group tension caused by my character stealing and possibly killing without provocation?

How do I tell my boss that I'm quitting in 15 days (a colleague left this week)

What should be the ideal length of sentences in a blog post for ease of reading?

What is the meaning of the following sentence?

When and why was runway 07/25 at Kai Tak removed?

Why is the principal energy of an electron lower for excited electrons in a higher energy state?

AWS Lambda: read csv file dimensions from an s3 bucket with Python without using Pandas or CSV package

Reading data from S3 using LambdaWhy is reading lines from stdin much slower in C++ than Python?Pandas writing dataframe to CSV fileUnicodeDecodeError when reading CSV file in Pandas with PythonHow do I install a Python package with a .whl file?how to read a file from folder structure in s3 bucket using pythonAWS Lambda and S3 and Pandas - Load CSV into S3, trigger Lambda, load into pandas, put back in bucket?How to reduce the size of packaged python zip files for AWS LambdaWrite csv file and save it into S3 using AWS Lambda (python)File name issue in AWS lambda s3 file unzip pythonLambda python 3.7 : Parameter validation failed:nInvalid type for parameter Dimensions[0]

0

good afternoon. I am hoping that someone can help me with this issue.

I have multiple CSV files that are sitting in an s3 folder. I would like to use python without the Pandas, and the csv package (because aws lambda has very limited packages available, and there is a size restriction) and loop through the files sitting in the s3 bucket, and read the csv dimensions (length of rows, and length of columns)

For example my s3 folder contains two csv files (1.csv, and 2 .csv)
my code will run through the specified s3 folder, and put the count of rows, and columns in 1 csv, and 2 csv, and puts the result in a new csv file. I greatly appreciate your help! I can do this using the Pandas package (thank god for Pandas, but aws lambda has restrictions that limits me on what I can use)

AWS lambda uses python 3.7

python pandas amazon-web-services amazon-s3 aws-lambda

share|improve this question

edited Mar 7 at 3:32

Haris Javed

asked Mar 7 at 3:17

Haris JavedHaris Javed

94

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  you know you can use pandas in AWS Lambda you just have to zip up the packaged dependency with the rest of your scripts
  
  – aws_apprentice
  Mar 7 at 3:17
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I think you forgot to set up permission for the Lambda function in the lambda dashboard. You need to make sure s3 buckets are accessable from Lambda. it is more a question about serverless and lambda than python
  
  – tim
  Mar 7 at 3:22
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  thank you sir aws_apprentice, I was also exploring that option. one of my co workers used that method, and he mentioned that we lose the ability to look into the code so I didn't explore that further, but I will explore it as well. That will be so much easier!
  
  – Haris Javed
  Mar 7 at 3:23
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Hi Tim good afternoon. My permissions are setup correctly, as I am able upload / remove files using boto3 in lambda. I can explore more around that to make sure nothing else is missing. Thank you
  
  – Haris Javed
  Mar 7 at 3:26
  

  
  
  
  

add a comment | 

0

good afternoon. I am hoping that someone can help me with this issue.

I have multiple CSV files that are sitting in an s3 folder. I would like to use python without the Pandas, and the csv package (because aws lambda has very limited packages available, and there is a size restriction) and loop through the files sitting in the s3 bucket, and read the csv dimensions (length of rows, and length of columns)

For example my s3 folder contains two csv files (1.csv, and 2 .csv)
my code will run through the specified s3 folder, and put the count of rows, and columns in 1 csv, and 2 csv, and puts the result in a new csv file. I greatly appreciate your help! I can do this using the Pandas package (thank god for Pandas, but aws lambda has restrictions that limits me on what I can use)

AWS lambda uses python 3.7

python pandas amazon-web-services amazon-s3 aws-lambda

share|improve this question

edited Mar 7 at 3:32

Haris Javed

asked Mar 7 at 3:17

Haris JavedHaris Javed

94

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  you know you can use pandas in AWS Lambda you just have to zip up the packaged dependency with the rest of your scripts
  
  – aws_apprentice
  Mar 7 at 3:17
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I think you forgot to set up permission for the Lambda function in the lambda dashboard. You need to make sure s3 buckets are accessable from Lambda. it is more a question about serverless and lambda than python
  
  – tim
  Mar 7 at 3:22
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  thank you sir aws_apprentice, I was also exploring that option. one of my co workers used that method, and he mentioned that we lose the ability to look into the code so I didn't explore that further, but I will explore it as well. That will be so much easier!
  
  – Haris Javed
  Mar 7 at 3:23
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Hi Tim good afternoon. My permissions are setup correctly, as I am able upload / remove files using boto3 in lambda. I can explore more around that to make sure nothing else is missing. Thank you
  
  – Haris Javed
  Mar 7 at 3:26
  

  
  
  
  

add a comment | 

good afternoon. I am hoping that someone can help me with this issue.

I have multiple CSV files that are sitting in an s3 folder. I would like to use python without the Pandas, and the csv package (because aws lambda has very limited packages available, and there is a size restriction) and loop through the files sitting in the s3 bucket, and read the csv dimensions (length of rows, and length of columns)

For example my s3 folder contains two csv files (1.csv, and 2 .csv)
my code will run through the specified s3 folder, and put the count of rows, and columns in 1 csv, and 2 csv, and puts the result in a new csv file. I greatly appreciate your help! I can do this using the Pandas package (thank god for Pandas, but aws lambda has restrictions that limits me on what I can use)

AWS lambda uses python 3.7

python pandas amazon-web-services amazon-s3 aws-lambda

share|improve this question

edited Mar 7 at 3:32

Haris Javed

asked Mar 7 at 3:17

Haris JavedHaris Javed

94

share|improve this question

edited Mar 7 at 3:32

Haris Javed

asked Mar 7 at 3:17

Haris JavedHaris Javed

94

share|improve this question

edited Mar 7 at 3:32

Haris Javed

asked Mar 7 at 3:17

Haris JavedHaris Javed

94

asked Mar 7 at 3:17

Haris JavedHaris Javed

94

asked Mar 7 at 3:17

Haris JavedHaris Javed

94

1 Answer
1

active

oldest

votes

0

If you can visit your s3 resources in your lambda function, then basically do this to check the rows,

def lambda_handler(event, context):
 import boto3 as bt3
 s3 = bt3.client('s3')
 csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
 csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')

 contents_1 = csv1_data['Body'].read()
 contents_2 = csv2_data['Body'].read()
 rows1 = contents_1.split()
 rows2=contents_2.split() 
 return len(rows1), len(rows2)

It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.

Regards.

share|improve this answer

edited Mar 8 at 3:01

answered Mar 7 at 22:05

timtim

873930

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @ Tim. Thank you so much Tim. This worked for my needs.
  
  – Haris Javed
  Mar 8 at 1:21
  
  

  
  
  
  

add a comment | 

Your Answer

StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55035518%2faws-lambda-read-csv-file-dimensions-from-an-s3-bucket-with-python-without-using%23new-answer', 'question_page');

);

Post as a guest

Name

Email

Required, but never shown

0

If you can visit your s3 resources in your lambda function, then basically do this to check the rows,

def lambda_handler(event, context):
 import boto3 as bt3
 s3 = bt3.client('s3')
 csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
 csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')

 contents_1 = csv1_data['Body'].read()
 contents_2 = csv2_data['Body'].read()
 rows1 = contents_1.split()
 rows2=contents_2.split() 
 return len(rows1), len(rows2)

It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.

Regards.

share|improve this answer

edited Mar 8 at 3:01

answered Mar 7 at 22:05

timtim

873930

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @ Tim. Thank you so much Tim. This worked for my needs.
  
  – Haris Javed
  Mar 8 at 1:21
  
  

  
  
  
  

add a comment | 

0

If you can visit your s3 resources in your lambda function, then basically do this to check the rows,

def lambda_handler(event, context):
 import boto3 as bt3
 s3 = bt3.client('s3')
 csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
 csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')

 contents_1 = csv1_data['Body'].read()
 contents_2 = csv2_data['Body'].read()
 rows1 = contents_1.split()
 rows2=contents_2.split() 
 return len(rows1), len(rows2)

It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.

Regards.

share|improve this answer

edited Mar 8 at 3:01

answered Mar 7 at 22:05

timtim

873930

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @ Tim. Thank you so much Tim. This worked for my needs.
  
  – Haris Javed
  Mar 8 at 1:21
  
  

  
  
  
  

add a comment | 

If you can visit your s3 resources in your lambda function, then basically do this to check the rows,

def lambda_handler(event, context):
 import boto3 as bt3
 s3 = bt3.client('s3')
 csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
 csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')

 contents_1 = csv1_data['Body'].read()
 contents_2 = csv2_data['Body'].read()
 rows1 = contents_1.split()
 rows2=contents_2.split() 
 return len(rows1), len(rows2)

It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.

Regards.

share|improve this answer

edited Mar 8 at 3:01

answered Mar 7 at 22:05

timtim

873930

def lambda_handler(event, context):
 import boto3 as bt3
 s3 = bt3.client('s3')
 csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
 csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')

 contents_1 = csv1_data['Body'].read()
 contents_2 = csv2_data['Body'].read()
 rows1 = contents_1.split()
 rows2=contents_2.split() 
 return len(rows1), len(rows2)

share|improve this answer

edited Mar 8 at 3:01

answered Mar 7 at 22:05

timtim

873930

answered Mar 7 at 22:05

timtim

873930

answered Mar 7 at 22:05

timtim

873930