How to Improve Data Scraping Using VBA?Options for HTML scraping?How to create Excel (.XLS and .XLSX) file in C# without installing Ms Office?Is there a way to crack the password on an Excel VBA Project?How to return a result from a VBA functionHow to avoid using Select in Excel VBAHeadless Browser and scraping - solutionsHow to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loopsData Scraping with td VBAscrape data VBA into excelUsing VBA to scrape data
What is it called when one voice type sings a 'solo'?
Manga about a female worker who got dragged into another world together with this high school girl and she was just told she's not needed anymore
Landlord wants to switch my lease to a "Land contract" to "get back at the city"
Is Social Media Science Fiction?
Why do we use polarized capacitors?
Why airport relocation isn't done gradually?
Denied boarding due to overcrowding, Sparpreis ticket. What are my rights?
Was there ever an axiom rendered a theorem?
Why did the Germans forbid the possession of pet pigeons in Rostov-on-Don in 1941?
Is every set a filtered colimit of finite sets?
aging parents with no investments
New order #4: World
Landing in very high winds
What is the meaning of "of trouble" in the following sentence?
How could a lack of term limits lead to a "dictatorship?"
Creating a loop after a break using Markov Chain in Tikz
Prime joint compound before latex paint?
Is there any use for defining additional entity types in a SOQL FROM clause?
Could a US political party gain complete control over the government by removing checks & balances?
How can I fix this gap between bookcases I made?
Is it legal to have the "// (c) 2019 John Smith" header in all files when there are hundreds of contributors?
Crop image to path created in TikZ?
Pristine Bit Checking
What do the Banks children have against barley water?
How to Improve Data Scraping Using VBA?
Options for HTML scraping?How to create Excel (.XLS and .XLSX) file in C# without installing Ms Office?Is there a way to crack the password on an Excel VBA Project?How to return a result from a VBA functionHow to avoid using Select in Excel VBAHeadless Browser and scraping - solutionsHow to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loopsData Scraping with td VBAscrape data VBA into excelUsing VBA to scrape data
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance.
Thanks In Advance
Note-I haven't posted URL as it is clients website. Sorry about that.
Sub FetchData()
Dim IE As Object
Dim Doc As HTMLDocument
Dim myStr As String
On Error Resume Next
Set IE = CreateObject("InternetExplorer.Application") 'SetBrowser
IE.Visible = False
IE.navigate "URL" 'Open website
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Set Doc = IE.Document
Doc.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
Doc.getElementById("txtPassword").Value = InputBox("Please Enter Your
Password")
Doc.getElementById("BtnLogin").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
IE.navigate "URL"
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Dim LastRow As Long
Set wks = ActiveSheet
LastRow = wks.Cells(wks.Rows.Count, "A").End(xlUp).Row
Set rowNo = wks.Range("A1:A" & LastRow)
For rowNo = 2 To LastRow
Doc.getElementById("txtField1").Value =
ThisWorkbook.Sheets("Sheet1").Range("A" & rowNo).Value
Doc.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
strVal1 = Doc.querySelectorAll("span")(33).innerText
ThisWorkbook.Sheets("Sheet1").Range("B" & rowNo).Value = strVal1
strVal2 = Doc.querySelectorAll("span")(35).innerText
ThisWorkbook.Sheets("Sheet1").Range("C" & rowNo).Value = strVal2
Next
End Sub
excel vba web-scraping
add a comment |
I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance.
Thanks In Advance
Note-I haven't posted URL as it is clients website. Sorry about that.
Sub FetchData()
Dim IE As Object
Dim Doc As HTMLDocument
Dim myStr As String
On Error Resume Next
Set IE = CreateObject("InternetExplorer.Application") 'SetBrowser
IE.Visible = False
IE.navigate "URL" 'Open website
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Set Doc = IE.Document
Doc.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
Doc.getElementById("txtPassword").Value = InputBox("Please Enter Your
Password")
Doc.getElementById("BtnLogin").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
IE.navigate "URL"
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Dim LastRow As Long
Set wks = ActiveSheet
LastRow = wks.Cells(wks.Rows.Count, "A").End(xlUp).Row
Set rowNo = wks.Range("A1:A" & LastRow)
For rowNo = 2 To LastRow
Doc.getElementById("txtField1").Value =
ThisWorkbook.Sheets("Sheet1").Range("A" & rowNo).Value
Doc.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
strVal1 = Doc.querySelectorAll("span")(33).innerText
ThisWorkbook.Sheets("Sheet1").Range("B" & rowNo).Value = strVal1
strVal2 = Doc.querySelectorAll("span")(35).innerText
ThisWorkbook.Sheets("Sheet1").Range("C" & rowNo).Value = strVal2
Next
End Sub
excel vba web-scraping
1
If your code is working, please consider posting to Code Review instead of Stack Overflow
– Sam
Mar 8 at 9:00
add a comment |
I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance.
Thanks In Advance
Note-I haven't posted URL as it is clients website. Sorry about that.
Sub FetchData()
Dim IE As Object
Dim Doc As HTMLDocument
Dim myStr As String
On Error Resume Next
Set IE = CreateObject("InternetExplorer.Application") 'SetBrowser
IE.Visible = False
IE.navigate "URL" 'Open website
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Set Doc = IE.Document
Doc.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
Doc.getElementById("txtPassword").Value = InputBox("Please Enter Your
Password")
Doc.getElementById("BtnLogin").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
IE.navigate "URL"
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Dim LastRow As Long
Set wks = ActiveSheet
LastRow = wks.Cells(wks.Rows.Count, "A").End(xlUp).Row
Set rowNo = wks.Range("A1:A" & LastRow)
For rowNo = 2 To LastRow
Doc.getElementById("txtField1").Value =
ThisWorkbook.Sheets("Sheet1").Range("A" & rowNo).Value
Doc.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
strVal1 = Doc.querySelectorAll("span")(33).innerText
ThisWorkbook.Sheets("Sheet1").Range("B" & rowNo).Value = strVal1
strVal2 = Doc.querySelectorAll("span")(35).innerText
ThisWorkbook.Sheets("Sheet1").Range("C" & rowNo).Value = strVal2
Next
End Sub
excel vba web-scraping
I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance.
Thanks In Advance
Note-I haven't posted URL as it is clients website. Sorry about that.
Sub FetchData()
Dim IE As Object
Dim Doc As HTMLDocument
Dim myStr As String
On Error Resume Next
Set IE = CreateObject("InternetExplorer.Application") 'SetBrowser
IE.Visible = False
IE.navigate "URL" 'Open website
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Set Doc = IE.Document
Doc.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
Doc.getElementById("txtPassword").Value = InputBox("Please Enter Your
Password")
Doc.getElementById("BtnLogin").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
IE.navigate "URL"
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
Dim LastRow As Long
Set wks = ActiveSheet
LastRow = wks.Cells(wks.Rows.Count, "A").End(xlUp).Row
Set rowNo = wks.Range("A1:A" & LastRow)
For rowNo = 2 To LastRow
Doc.getElementById("txtField1").Value =
ThisWorkbook.Sheets("Sheet1").Range("A" & rowNo).Value
Doc.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop
strVal1 = Doc.querySelectorAll("span")(33).innerText
ThisWorkbook.Sheets("Sheet1").Range("B" & rowNo).Value = strVal1
strVal2 = Doc.querySelectorAll("span")(35).innerText
ThisWorkbook.Sheets("Sheet1").Range("C" & rowNo).Value = strVal2
Next
End Sub
excel vba web-scraping
excel vba web-scraping
edited Mar 8 at 9:23
Pᴇʜ
25.2k63052
25.2k63052
asked Mar 8 at 7:18
Murali KMurali K
11
11
1
If your code is working, please consider posting to Code Review instead of Stack Overflow
– Sam
Mar 8 at 9:00
add a comment |
1
If your code is working, please consider posting to Code Review instead of Stack Overflow
– Sam
Mar 8 at 9:00
1
1
If your code is working, please consider posting to Code Review instead of Stack Overflow
– Sam
Mar 8 at 9:00
If your code is working, please consider posting to Code Review instead of Stack Overflow
– Sam
Mar 8 at 9:00
add a comment |
1 Answer
1
active
oldest
votes
Can't guarantee this will run. Points to note:
- Use of
Worksheets
collection - Use of
Option Explicit
- this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range. - Removal of
On Error Resume Next
- Putting all worksheets into variables
- Placement of values into array and looping array to get id values. Looping sheet is expensive
- Use of early binding and adding class to InternetExplorer
- Assumption that after login a new url present and that you need to navigate back to that before each new loop value
- Removal of hungarian notation
- Ids are fastest selector method so no improvement there
- With your css type selectors, e.g.
.document.querySelectorAll("span")(33)
, you might seek whether there is a single node short selector that can be used, rather than using nodeList
VBA:
Option Explicit
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet
Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)
With ie
.Visible = False
.Navigate2 "URL" 'Open website
While .Busy Or .readyState < 4: DoEvents: Wend
.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click
While .Busy Or ie.readyState < 4: DoEvents: Wend
Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL
For i = LBound(loopvalues) To UBound(loopvalues)
.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
While .Busy Or .readyState < 4: DoEvents: Wend
val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2
.Navigate2 newURL
While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub
Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?
– Murali K
Mar 8 at 9:53
You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.
– QHarr
Mar 8 at 9:58
its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?
– Murali K
Mar 8 at 10:28
Where i say .navigate2 newUrl
– QHarr
Mar 8 at 10:30
Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"
– Murali K
Mar 8 at 11:41
|
show 17 more comments
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55058469%2fhow-to-improve-data-scraping-using-vba%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Can't guarantee this will run. Points to note:
- Use of
Worksheets
collection - Use of
Option Explicit
- this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range. - Removal of
On Error Resume Next
- Putting all worksheets into variables
- Placement of values into array and looping array to get id values. Looping sheet is expensive
- Use of early binding and adding class to InternetExplorer
- Assumption that after login a new url present and that you need to navigate back to that before each new loop value
- Removal of hungarian notation
- Ids are fastest selector method so no improvement there
- With your css type selectors, e.g.
.document.querySelectorAll("span")(33)
, you might seek whether there is a single node short selector that can be used, rather than using nodeList
VBA:
Option Explicit
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet
Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)
With ie
.Visible = False
.Navigate2 "URL" 'Open website
While .Busy Or .readyState < 4: DoEvents: Wend
.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click
While .Busy Or ie.readyState < 4: DoEvents: Wend
Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL
For i = LBound(loopvalues) To UBound(loopvalues)
.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
While .Busy Or .readyState < 4: DoEvents: Wend
val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2
.Navigate2 newURL
While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub
Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?
– Murali K
Mar 8 at 9:53
You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.
– QHarr
Mar 8 at 9:58
its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?
– Murali K
Mar 8 at 10:28
Where i say .navigate2 newUrl
– QHarr
Mar 8 at 10:30
Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"
– Murali K
Mar 8 at 11:41
|
show 17 more comments
Can't guarantee this will run. Points to note:
- Use of
Worksheets
collection - Use of
Option Explicit
- this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range. - Removal of
On Error Resume Next
- Putting all worksheets into variables
- Placement of values into array and looping array to get id values. Looping sheet is expensive
- Use of early binding and adding class to InternetExplorer
- Assumption that after login a new url present and that you need to navigate back to that before each new loop value
- Removal of hungarian notation
- Ids are fastest selector method so no improvement there
- With your css type selectors, e.g.
.document.querySelectorAll("span")(33)
, you might seek whether there is a single node short selector that can be used, rather than using nodeList
VBA:
Option Explicit
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet
Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)
With ie
.Visible = False
.Navigate2 "URL" 'Open website
While .Busy Or .readyState < 4: DoEvents: Wend
.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click
While .Busy Or ie.readyState < 4: DoEvents: Wend
Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL
For i = LBound(loopvalues) To UBound(loopvalues)
.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
While .Busy Or .readyState < 4: DoEvents: Wend
val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2
.Navigate2 newURL
While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub
Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?
– Murali K
Mar 8 at 9:53
You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.
– QHarr
Mar 8 at 9:58
its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?
– Murali K
Mar 8 at 10:28
Where i say .navigate2 newUrl
– QHarr
Mar 8 at 10:30
Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"
– Murali K
Mar 8 at 11:41
|
show 17 more comments
Can't guarantee this will run. Points to note:
- Use of
Worksheets
collection - Use of
Option Explicit
- this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range. - Removal of
On Error Resume Next
- Putting all worksheets into variables
- Placement of values into array and looping array to get id values. Looping sheet is expensive
- Use of early binding and adding class to InternetExplorer
- Assumption that after login a new url present and that you need to navigate back to that before each new loop value
- Removal of hungarian notation
- Ids are fastest selector method so no improvement there
- With your css type selectors, e.g.
.document.querySelectorAll("span")(33)
, you might seek whether there is a single node short selector that can be used, rather than using nodeList
VBA:
Option Explicit
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet
Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)
With ie
.Visible = False
.Navigate2 "URL" 'Open website
While .Busy Or .readyState < 4: DoEvents: Wend
.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click
While .Busy Or ie.readyState < 4: DoEvents: Wend
Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL
For i = LBound(loopvalues) To UBound(loopvalues)
.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
While .Busy Or .readyState < 4: DoEvents: Wend
val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2
.Navigate2 newURL
While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub
Can't guarantee this will run. Points to note:
- Use of
Worksheets
collection - Use of
Option Explicit
- this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range. - Removal of
On Error Resume Next
- Putting all worksheets into variables
- Placement of values into array and looping array to get id values. Looping sheet is expensive
- Use of early binding and adding class to InternetExplorer
- Assumption that after login a new url present and that you need to navigate back to that before each new loop value
- Removal of hungarian notation
- Ids are fastest selector method so no improvement there
- With your css type selectors, e.g.
.document.querySelectorAll("span")(33)
, you might seek whether there is a single node short selector that can be used, rather than using nodeList
VBA:
Option Explicit
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet
Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)
With ie
.Visible = False
.Navigate2 "URL" 'Open website
While .Busy Or .readyState < 4: DoEvents: Wend
.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click
While .Busy Or ie.readyState < 4: DoEvents: Wend
Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL
For i = LBound(loopvalues) To UBound(loopvalues)
.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
While .Busy Or .readyState < 4: DoEvents: Wend
val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2
.Navigate2 newURL
While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub
edited Mar 8 at 15:23
answered Mar 8 at 8:58
QHarrQHarr
37.4k82244
37.4k82244
Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?
– Murali K
Mar 8 at 9:53
You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.
– QHarr
Mar 8 at 9:58
its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?
– Murali K
Mar 8 at 10:28
Where i say .navigate2 newUrl
– QHarr
Mar 8 at 10:30
Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"
– Murali K
Mar 8 at 11:41
|
show 17 more comments
Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?
– Murali K
Mar 8 at 9:53
You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.
– QHarr
Mar 8 at 9:58
its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?
– Murali K
Mar 8 at 10:28
Where i say .navigate2 newUrl
– QHarr
Mar 8 at 10:30
Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"
– Murali K
Mar 8 at 11:41
Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?
– Murali K
Mar 8 at 9:53
Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?
– Murali K
Mar 8 at 9:53
You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.
– QHarr
Mar 8 at 9:58
You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.
– QHarr
Mar 8 at 9:58
its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?
– Murali K
Mar 8 at 10:28
its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?
– Murali K
Mar 8 at 10:28
Where i say .navigate2 newUrl
– QHarr
Mar 8 at 10:30
Where i say .navigate2 newUrl
– QHarr
Mar 8 at 10:30
Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"
– Murali K
Mar 8 at 11:41
Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"
– Murali K
Mar 8 at 11:41
|
show 17 more comments
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55058469%2fhow-to-improve-data-scraping-using-vba%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
If your code is working, please consider posting to Code Review instead of Stack Overflow
– Sam
Mar 8 at 9:00