Pandas Dataframe to Postgres table conversion not working2019 Community Moderator ElectionPostgreSQL “DESCRIBE TABLE”Show tables in PostgreSQLSelecting multiple columns in a pandas dataframeRenaming columns in pandasAdding new column to existing DataFrame in Python pandasDelete column from pandas DataFrame by column name“Large data” work flows using pandasHow to iterate over rows in a DataFrame in Pandas?Select rows from a DataFrame based on values in a column in pandasGet list from pandas DataFrame column headers
Is there any common country to visit for uk and schengen visa?
Output visual diagram of picture
Do people actually use the word "kaputt" in conversation?
Is xar preinstalled on macOS?
Writing in a Christian voice
What kind of footwear is suitable for walking in micro gravity environment?
Why is there so much iron?
PTIJ: Where did Achashverosh's years wander off to?
Homology of the fiber
I got the following comment from a reputed math journal. What does it mean?
Does fire aspect on a sword, destroy mob drops?
Can other pieces capture a threatening piece and prevent a checkmate?
Why are there no stars visible in cislunar space?
Knife as defense against stray dogs
Can "few" be used as a subject? If so, what is the rule?
Why didn’t Eve recognize the little cockroach as a living organism?
When should a starting writer get his own webpage?
Exit shell with shortcut (not typing exit) that closes session properly
How do researchers send unsolicited emails asking for feedback on their works?
Turning a hard to access nut?
Why do I have a large white artefact on the rendered image?
How to find the largest number(s) in a list of elements, possibly non-unique?
How to test the sharpness of a knife?
is this saw blade faulty?
Pandas Dataframe to Postgres table conversion not working
2019 Community Moderator ElectionPostgreSQL “DESCRIBE TABLE”Show tables in PostgreSQLSelecting multiple columns in a pandas dataframeRenaming columns in pandasAdding new column to existing DataFrame in Python pandasDelete column from pandas DataFrame by column name“Large data” work flows using pandasHow to iterate over rows in a DataFrame in Pandas?Select rows from a DataFrame based on values in a column in pandasGet list from pandas DataFrame column headers
I am converting a csv file into a Pandas dataframe and then converting it to Postgres table essentially.
The problem is that I am able to create a table in Postgres but I am unable to select column names from the table while querying it.
This is the sample code I have:
import pandas as pd
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql://postgres:pwd@localhost:5432/test')
def convertcsvtopostgres(csvfileloc, table_name, delimiter):
data = pd.read_csv(csvfileloc, sep=delimiter, encoding='latin-1')
data.head()
data1 = data.rename(columns=lambda x: x.strip())
data1.to_sql(table_name, engine, index=False)
convertcsvtopostgres("Product.csv","t_product","~")
I can do a select * from test.t_product; but I am unable to do a select product_id from test.t_product;
I am not sure if that is happening because of the encoding of the file and the conversion because of that. Is there any way around this, since I do not want to specify the table structure each time.
python pandas postgresql psycopg2
asked Mar 6 at 23:37
Manas JaniManas Jani
301319
add a comment |
I am converting a csv file into a Pandas dataframe and then converting it to Postgres table essentially.
The problem is that I am able to create a table in Postgres but I am unable to select column names from the table while querying it.
This is the sample code I have:
import pandas as pd
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql://postgres:pwd@localhost:5432/test')
def convertcsvtopostgres(csvfileloc, table_name, delimiter):
data = pd.read_csv(csvfileloc, sep=delimiter, encoding='latin-1')
data.head()
data1 = data.rename(columns=lambda x: x.strip())
data1.to_sql(table_name, engine, index=False)
convertcsvtopostgres("Product.csv","t_product","~")
I can do a select * from test.t_product; but I am unable to do a select product_id from test.t_product;
I am not sure if that is happening because of the encoding of the file and the conversion because of that. Is there any way around this, since I do not want to specify the table structure each time.
python pandas postgresql psycopg2
asked Mar 6 at 23:37
Manas JaniManas Jani
301319
At apsqlprompt, typed test.t_product. That should show you the names of the columns in thetest.t_producttable. On the Python side, you couldprint(data1.columns.tolist())to see what Python thinks the column names are.
– unutbu
Mar 7 at 2:58
If that does not clarify what the issue is, please post a few lines of the csv file. That might help us reproduce the problem.
– unutbu
Mar 7 at 3:00
Do you have to use python/pandas? Otherwise, getting a CSV into a PostgreSQL table is supported out-of-the-box using COPY
– Ancoron
Mar 7 at 12:51
No, its not necessary for me to use pandas/python. I just used it because I thought it would be better fit with the encoding piece of the problem that I have with the csv files.
– Manas Jani
Mar 7 at 15:58
add a comment |
I am converting a csv file into a Pandas dataframe and then converting it to Postgres table essentially.
The problem is that I am able to create a table in Postgres but I am unable to select column names from the table while querying it.
This is the sample code I have:
import pandas as pd
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql://postgres:pwd@localhost:5432/test')
def convertcsvtopostgres(csvfileloc, table_name, delimiter):
data = pd.read_csv(csvfileloc, sep=delimiter, encoding='latin-1')
data.head()
data1 = data.rename(columns=lambda x: x.strip())
data1.to_sql(table_name, engine, index=False)
convertcsvtopostgres("Product.csv","t_product","~")
I can do a select * from test.t_product; but I am unable to do a select product_id from test.t_product;
I am not sure if that is happening because of the encoding of the file and the conversion because of that. Is there any way around this, since I do not want to specify the table structure each time.
python pandas postgresql psycopg2
asked Mar 6 at 23:37
Manas JaniManas Jani
301319
I am converting a csv file into a Pandas dataframe and then converting it to Postgres table essentially.
The problem is that I am able to create a table in Postgres but I am unable to select column names from the table while querying it.
This is the sample code I have:
import pandas as pd
from sqlalchemy import create_engine
import psycopg2
engine = create_engine('postgresql://postgres:pwd@localhost:5432/test')
def convertcsvtopostgres(csvfileloc, table_name, delimiter):
data = pd.read_csv(csvfileloc, sep=delimiter, encoding='latin-1')
data.head()
data1 = data.rename(columns=lambda x: x.strip())
data1.to_sql(table_name, engine, index=False)
convertcsvtopostgres("Product.csv","t_product","~")
I can do a select * from test.t_product; but I am unable to do a select product_id from test.t_product;
I am not sure if that is happening because of the encoding of the file and the conversion because of that. Is there any way around this, since I do not want to specify the table structure each time.
python pandas postgresql psycopg2
python pandas postgresql psycopg2
asked Mar 6 at 23:37
Manas JaniManas Jani
301319
asked Mar 6 at 23:37
Manas JaniManas Jani
301319
asked Mar 6 at 23:37
Manas JaniManas Jani
301319
asked Mar 6 at 23:37
Manas JaniManas Jani
301319
asked Mar 6 at 23:37
Manas JaniManas Jani
301319
301319
At apsqlprompt, typed test.t_product. That should show you the names of the columns in thetest.t_producttable. On the Python side, you couldprint(data1.columns.tolist())to see what Python thinks the column names are.
– unutbu
Mar 7 at 2:58
If that does not clarify what the issue is, please post a few lines of the csv file. That might help us reproduce the problem.
– unutbu
Mar 7 at 3:00
Do you have to use python/pandas? Otherwise, getting a CSV into a PostgreSQL table is supported out-of-the-box using COPY
– Ancoron
Mar 7 at 12:51
No, its not necessary for me to use pandas/python. I just used it because I thought it would be better fit with the encoding piece of the problem that I have with the csv files.
– Manas Jani
Mar 7 at 15:58
add a comment |
At apsqlprompt, typed test.t_product. That should show you the names of the columns in thetest.t_producttable. On the Python side, you couldprint(data1.columns.tolist())to see what Python thinks the column names are.
– unutbu
Mar 7 at 2:58
If that does not clarify what the issue is, please post a few lines of the csv file. That might help us reproduce the problem.
– unutbu
Mar 7 at 3:00
Do you have to use python/pandas? Otherwise, getting a CSV into a PostgreSQL table is supported out-of-the-box using COPY
– Ancoron
Mar 7 at 12:51
No, its not necessary for me to use pandas/python. I just used it because I thought it would be better fit with the encoding piece of the problem that I have with the csv files.
– Manas Jani
Mar 7 at 15:58
At a
psql prompt, type d test.t_product. That should show you the names of the columns in the test.t_product table. On the Python side, you could print(data1.columns.tolist()) to see what Python thinks the column names are.– unutbu
Mar 7 at 2:58
At a
psql prompt, type d test.t_product. That should show you the names of the columns in the test.t_product table. On the Python side, you could print(data1.columns.tolist()) to see what Python thinks the column names are.– unutbu
Mar 7 at 2:58
If that does not clarify what the issue is, please post a few lines of the csv file. That might help us reproduce the problem.
– unutbu
Mar 7 at 3:00
If that does not clarify what the issue is, please post a few lines of the csv file. That might help us reproduce the problem.
– unutbu
Mar 7 at 3:00
Do you have to use python/pandas? Otherwise, getting a CSV into a PostgreSQL table is supported out-of-the-box using COPY
– Ancoron
Mar 7 at 12:51
Do you have to use python/pandas? Otherwise, getting a CSV into a PostgreSQL table is supported out-of-the-box using COPY
– Ancoron
Mar 7 at 12:51
No, its not necessary for me to use pandas/python. I just used it because I thought it would be better fit with the encoding piece of the problem that I have with the csv files.
– Manas Jani
Mar 7 at 15:58
No, its not necessary for me to use pandas/python. I just used it because I thought it would be better fit with the encoding piece of the problem that I have with the csv files.
– Manas Jani
Mar 7 at 15:58
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55033886%2fpandas-dataframe-to-postgres-table-conversion-not-working%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55033886%2fpandas-dataframe-to-postgres-table-conversion-not-working%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
At a
psqlprompt, typed test.t_product. That should show you the names of the columns in thetest.t_producttable. On the Python side, you couldprint(data1.columns.tolist())to see what Python thinks the column names are.– unutbu
Mar 7 at 2:58
If that does not clarify what the issue is, please post a few lines of the csv file. That might help us reproduce the problem.
– unutbu
Mar 7 at 3:00
Do you have to use python/pandas? Otherwise, getting a CSV into a PostgreSQL table is supported out-of-the-box using COPY
– Ancoron
Mar 7 at 12:51
No, its not necessary for me to use pandas/python. I just used it because I thought it would be better fit with the encoding piece of the problem that I have with the csv files.
– Manas Jani
Mar 7 at 15:58