AWS Lambda: read csv file dimensions from an s3 bucket with Python without using Pandas or CSV packageReading data from S3 using LambdaWhy is reading lines from stdin much slower in C++ than Python?Pandas writing dataframe to CSV fileUnicodeDecodeError when reading CSV file in Pandas with PythonHow do I install a Python package with a .whl file?how to read a file from folder structure in s3 bucket using pythonAWS Lambda and S3 and Pandas - Load CSV into S3, trigger Lambda, load into pandas, put back in bucket?How to reduce the size of packaged python zip files for AWS LambdaWrite csv file and save it into S3 using AWS Lambda (python)File name issue in AWS lambda s3 file unzip pythonLambda python 3.7 : Parameter validation failed:nInvalid type for parameter Dimensions[0]
Has the laser at Magurele, Romania reached a tenth of the Sun's power?
Giving feedback to someone without sounding prejudiced
How do you justify more code being written by following clean code practices?
Is there a distance limit for minecart tracks?
If the only attacker is removed from combat, is a creature still counted as having attacked this turn?
Ways of geometrical multiplication
Would this string work as string?
What does "Scientists rise up against statistical significance" mean? (Comment in Nature)
Why didn’t Eve recognize the little cockroach as a living organism?
How can I, as DM, avoid the Conga Line of Death occurring when implementing some form of flanking rule?
What's the name of the logical fallacy where a debater extends a statement far beyond the original statement to make it true?
How to make money from a browser who sees 5 seconds into the future of any web page?
What is the meaning of "You've never met a graph you didn't like?"
Can I run 125kHz RF circuit on a breadboard?
Does the Crossbow Expert feat's extra crossbow attack work with the reaction attack from a Hunter ranger's Giant Killer feature?
"Oh no!" in Latin
Why is participating in the European Parliamentary elections used as a threat?
Personal or impersonal in a technical resume
How do I fix the group tension caused by my character stealing and possibly killing without provocation?
How do I tell my boss that I'm quitting in 15 days (a colleague left this week)
What should be the ideal length of sentences in a blog post for ease of reading?
What is the meaning of the following sentence?
When and why was runway 07/25 at Kai Tak removed?
Why is the principal energy of an electron lower for excited electrons in a higher energy state?
AWS Lambda: read csv file dimensions from an s3 bucket with Python without using Pandas or CSV package
Reading data from S3 using LambdaWhy is reading lines from stdin much slower in C++ than Python?Pandas writing dataframe to CSV fileUnicodeDecodeError when reading CSV file in Pandas with PythonHow do I install a Python package with a .whl file?how to read a file from folder structure in s3 bucket using pythonAWS Lambda and S3 and Pandas - Load CSV into S3, trigger Lambda, load into pandas, put back in bucket?How to reduce the size of packaged python zip files for AWS LambdaWrite csv file and save it into S3 using AWS Lambda (python)File name issue in AWS lambda s3 file unzip pythonLambda python 3.7 : Parameter validation failed:nInvalid type for parameter Dimensions[0]
good afternoon. I am hoping that someone can help me with this issue.
I have multiple CSV files that are sitting in an s3 folder. I would like to use python without the Pandas, and the csv package (because aws lambda has very limited packages available, and there is a size restriction) and loop through the files sitting in the s3 bucket, and read the csv dimensions (length of rows, and length of columns)
For example my s3 folder contains two csv files (1.csv, and 2 .csv)
my code will run through the specified s3 folder, and put the count of rows, and columns in 1 csv, and 2 csv, and puts the result in a new csv file. I greatly appreciate your help! I can do this using the Pandas package (thank god for Pandas, but aws lambda has restrictions that limits me on what I can use)
AWS lambda uses python 3.7
python pandas amazon-web-services amazon-s3 aws-lambda
asked Mar 7 at 3:17
Haris JavedHaris Javed
94
add a comment |
good afternoon. I am hoping that someone can help me with this issue.
I have multiple CSV files that are sitting in an s3 folder. I would like to use python without the Pandas, and the csv package (because aws lambda has very limited packages available, and there is a size restriction) and loop through the files sitting in the s3 bucket, and read the csv dimensions (length of rows, and length of columns)
For example my s3 folder contains two csv files (1.csv, and 2 .csv)
my code will run through the specified s3 folder, and put the count of rows, and columns in 1 csv, and 2 csv, and puts the result in a new csv file. I greatly appreciate your help! I can do this using the Pandas package (thank god for Pandas, but aws lambda has restrictions that limits me on what I can use)
AWS lambda uses python 3.7
python pandas amazon-web-services amazon-s3 aws-lambda
asked Mar 7 at 3:17
Haris JavedHaris Javed
94
you know you can usepandasin AWS Lambda you just have to zip up the packaged dependency with the rest of your scripts
– aws_apprentice
Mar 7 at 3:17
I think you forgot to set up permission for theLambda functionin the lambda dashboard. You need to make sure s3 buckets are accessable fromLambda. it is more a question aboutserverlessandlambdathanpython
– tim
Mar 7 at 3:22
thank you sir aws_apprentice, I was also exploring that option. one of my co workers used that method, and he mentioned that we lose the ability to look into the code so I didn't explore that further, but I will explore it as well. That will be so much easier!
– Haris Javed
Mar 7 at 3:23
Hi Tim good afternoon. My permissions are setup correctly, as I am able upload / remove files using boto3 in lambda. I can explore more around that to make sure nothing else is missing. Thank you
– Haris Javed
Mar 7 at 3:26
add a comment |
good afternoon. I am hoping that someone can help me with this issue.
I have multiple CSV files that are sitting in an s3 folder. I would like to use python without the Pandas, and the csv package (because aws lambda has very limited packages available, and there is a size restriction) and loop through the files sitting in the s3 bucket, and read the csv dimensions (length of rows, and length of columns)
For example my s3 folder contains two csv files (1.csv, and 2 .csv)
my code will run through the specified s3 folder, and put the count of rows, and columns in 1 csv, and 2 csv, and puts the result in a new csv file. I greatly appreciate your help! I can do this using the Pandas package (thank god for Pandas, but aws lambda has restrictions that limits me on what I can use)
AWS lambda uses python 3.7
python pandas amazon-web-services amazon-s3 aws-lambda
asked Mar 7 at 3:17
Haris JavedHaris Javed
94
good afternoon. I am hoping that someone can help me with this issue.
I have multiple CSV files that are sitting in an s3 folder. I would like to use python without the Pandas, and the csv package (because aws lambda has very limited packages available, and there is a size restriction) and loop through the files sitting in the s3 bucket, and read the csv dimensions (length of rows, and length of columns)
For example my s3 folder contains two csv files (1.csv, and 2 .csv)
my code will run through the specified s3 folder, and put the count of rows, and columns in 1 csv, and 2 csv, and puts the result in a new csv file. I greatly appreciate your help! I can do this using the Pandas package (thank god for Pandas, but aws lambda has restrictions that limits me on what I can use)
AWS lambda uses python 3.7
python pandas amazon-web-services amazon-s3 aws-lambda
python pandas amazon-web-services amazon-s3 aws-lambda
asked Mar 7 at 3:17
Haris JavedHaris Javed
94
asked Mar 7 at 3:17
Haris JavedHaris Javed
94
edited Mar 7 at 3:32
Haris Javed
asked Mar 7 at 3:17
Haris JavedHaris Javed
94
asked Mar 7 at 3:17
Haris JavedHaris Javed
94
asked Mar 7 at 3:17
Haris JavedHaris Javed
94
94
you know you can usepandasin AWS Lambda you just have to zip up the packaged dependency with the rest of your scripts
– aws_apprentice
Mar 7 at 3:17
I think you forgot to set up permission for theLambda functionin the lambda dashboard. You need to make sure s3 buckets are accessable fromLambda. it is more a question aboutserverlessandlambdathanpython
– tim
Mar 7 at 3:22
thank you sir aws_apprentice, I was also exploring that option. one of my co workers used that method, and he mentioned that we lose the ability to look into the code so I didn't explore that further, but I will explore it as well. That will be so much easier!
– Haris Javed
Mar 7 at 3:23
Hi Tim good afternoon. My permissions are setup correctly, as I am able upload / remove files using boto3 in lambda. I can explore more around that to make sure nothing else is missing. Thank you
– Haris Javed
Mar 7 at 3:26
add a comment |
you know you can usepandasin AWS Lambda you just have to zip up the packaged dependency with the rest of your scripts
– aws_apprentice
Mar 7 at 3:17
I think you forgot to set up permission for theLambda functionin the lambda dashboard. You need to make sure s3 buckets are accessable fromLambda. it is more a question aboutserverlessandlambdathanpython
– tim
Mar 7 at 3:22
thank you sir aws_apprentice, I was also exploring that option. one of my co workers used that method, and he mentioned that we lose the ability to look into the code so I didn't explore that further, but I will explore it as well. That will be so much easier!
– Haris Javed
Mar 7 at 3:23
Hi Tim good afternoon. My permissions are setup correctly, as I am able upload / remove files using boto3 in lambda. I can explore more around that to make sure nothing else is missing. Thank you
– Haris Javed
Mar 7 at 3:26
you know you can use
pandas in AWS Lambda you just have to zip up the packaged dependency with the rest of your scripts– aws_apprentice
Mar 7 at 3:17
you know you can use
pandas in AWS Lambda you just have to zip up the packaged dependency with the rest of your scripts– aws_apprentice
Mar 7 at 3:17
I think you forgot to set up permission for the
Lambda function in the lambda dashboard. You need to make sure s3 buckets are accessable from Lambda. it is more a question about serverless and lambda than python– tim
Mar 7 at 3:22
I think you forgot to set up permission for the
Lambda function in the lambda dashboard. You need to make sure s3 buckets are accessable from Lambda. it is more a question about serverless and lambda than python– tim
Mar 7 at 3:22
thank you sir aws_apprentice, I was also exploring that option. one of my co workers used that method, and he mentioned that we lose the ability to look into the code so I didn't explore that further, but I will explore it as well. That will be so much easier!
– Haris Javed
Mar 7 at 3:23
thank you sir aws_apprentice, I was also exploring that option. one of my co workers used that method, and he mentioned that we lose the ability to look into the code so I didn't explore that further, but I will explore it as well. That will be so much easier!
– Haris Javed
Mar 7 at 3:23
Hi Tim good afternoon. My permissions are setup correctly, as I am able upload / remove files using boto3 in lambda. I can explore more around that to make sure nothing else is missing. Thank you
– Haris Javed
Mar 7 at 3:26
Hi Tim good afternoon. My permissions are setup correctly, as I am able upload / remove files using boto3 in lambda. I can explore more around that to make sure nothing else is missing. Thank you
– Haris Javed
Mar 7 at 3:26
add a comment |
1 Answer
1
active
oldest
votes
If you can visit your s3 resources in your lambda function, then basically do this to check the rows,
def lambda_handler(event, context):
import boto3 as bt3
s3 = bt3.client('s3')
csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')
contents_1 = csv1_data['Body'].read()
contents_2 = csv2_data['Body'].read()
rows1 = contents_1.split()
rows2=contents_2.split()
return len(rows1), len(rows2)
It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.
Regards.
answered Mar 7 at 22:05
timtim
873930
@ Tim. Thank you so much Tim. This worked for my needs.
– Haris Javed
Mar 8 at 1:21
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55035518%2faws-lambda-read-csv-file-dimensions-from-an-s3-bucket-with-python-without-using%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
If you can visit your s3 resources in your lambda function, then basically do this to check the rows,
def lambda_handler(event, context):
import boto3 as bt3
s3 = bt3.client('s3')
csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')
contents_1 = csv1_data['Body'].read()
contents_2 = csv2_data['Body'].read()
rows1 = contents_1.split()
rows2=contents_2.split()
return len(rows1), len(rows2)
It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.
Regards.
answered Mar 7 at 22:05
timtim
873930
@ Tim. Thank you so much Tim. This worked for my needs.
– Haris Javed
Mar 8 at 1:21
add a comment |
If you can visit your s3 resources in your lambda function, then basically do this to check the rows,
def lambda_handler(event, context):
import boto3 as bt3
s3 = bt3.client('s3')
csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')
contents_1 = csv1_data['Body'].read()
contents_2 = csv2_data['Body'].read()
rows1 = contents_1.split()
rows2=contents_2.split()
return len(rows1), len(rows2)
It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.
Regards.
answered Mar 7 at 22:05
timtim
873930
@ Tim. Thank you so much Tim. This worked for my needs.
– Haris Javed
Mar 8 at 1:21
add a comment |
If you can visit your s3 resources in your lambda function, then basically do this to check the rows,
def lambda_handler(event, context):
import boto3 as bt3
s3 = bt3.client('s3')
csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')
contents_1 = csv1_data['Body'].read()
contents_2 = csv2_data['Body'].read()
rows1 = contents_1.split()
rows2=contents_2.split()
return len(rows1), len(rows2)
It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.
Regards.
answered Mar 7 at 22:05
timtim
873930
If you can visit your s3 resources in your lambda function, then basically do this to check the rows,
def lambda_handler(event, context):
import boto3 as bt3
s3 = bt3.client('s3')
csv1_data = s3.get_object(Bucket='the_s3_bucket', Key='1.csv')
csv2_data = s3.get_object(Bucket='the_s3_bucket', Key='2.csv')
contents_1 = csv1_data['Body'].read()
contents_2 = csv2_data['Body'].read()
rows1 = contents_1.split()
rows2=contents_2.split()
return len(rows1), len(rows2)
It should work directly, if not, please let me know. BTW, hard coding the bucket and file name into the function like what I did in the sample is not a good idea at all.
Regards.
answered Mar 7 at 22:05
timtim
873930
edited Mar 8 at 3:01
answered Mar 7 at 22:05
timtim
873930
answered Mar 7 at 22:05
timtim
873930
answered Mar 7 at 22:05
timtim
873930
873930
@ Tim. Thank you so much Tim. This worked for my needs.
– Haris Javed
Mar 8 at 1:21
add a comment |
@ Tim. Thank you so much Tim. This worked for my needs.
– Haris Javed
Mar 8 at 1:21
@ Tim. Thank you so much Tim. This worked for my needs.
– Haris Javed
Mar 8 at 1:21
@ Tim. Thank you so much Tim. This worked for my needs.
– Haris Javed
Mar 8 at 1:21
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55035518%2faws-lambda-read-csv-file-dimensions-from-an-s3-bucket-with-python-without-using%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
you know you can use
pandasin AWS Lambda you just have to zip up the packaged dependency with the rest of your scripts– aws_apprentice
Mar 7 at 3:17
I think you forgot to set up permission for the
Lambda functionin the lambda dashboard. You need to make sure s3 buckets are accessable fromLambda. it is more a question aboutserverlessandlambdathanpython– tim
Mar 7 at 3:22
thank you sir aws_apprentice, I was also exploring that option. one of my co workers used that method, and he mentioned that we lose the ability to look into the code so I didn't explore that further, but I will explore it as well. That will be so much easier!
– Haris Javed
Mar 7 at 3:23
Hi Tim good afternoon. My permissions are setup correctly, as I am able upload / remove files using boto3 in lambda. I can explore more around that to make sure nothing else is missing. Thank you
– Haris Javed
Mar 7 at 3:26