How to Improve Data Scraping Using VBA?Options for HTML scraping?How to create Excel (.XLS and .XLSX) file in C# without installing Ms Office?Is there a way to crack the password on an Excel VBA Project?How to return a result from a VBA functionHow to avoid using Select in Excel VBAHeadless Browser and scraping - solutionsHow to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loopsData Scraping with td VBAscrape data VBA into excelUsing VBA to scrape data

What is it called when one voice type sings a 'solo'?

Manga about a female worker who got dragged into another world together with this high school girl and she was just told she's not needed anymore

Landlord wants to switch my lease to a "Land contract" to "get back at the city"

Is Social Media Science Fiction?

Why do we use polarized capacitors?

Why airport relocation isn't done gradually?

Denied boarding due to overcrowding, Sparpreis ticket. What are my rights?

Was there ever an axiom rendered a theorem?

Why did the Germans forbid the possession of pet pigeons in Rostov-on-Don in 1941?

Is every set a filtered colimit of finite sets?

aging parents with no investments

New order #4: World

Landing in very high winds

What is the meaning of "of trouble" in the following sentence?

How could a lack of term limits lead to a "dictatorship?"

Creating a loop after a break using Markov Chain in Tikz

Prime joint compound before latex paint?

Is there any use for defining additional entity types in a SOQL FROM clause?

Could a US political party gain complete control over the government by removing checks & balances?

How can I fix this gap between bookcases I made?

Is it legal to have the "// (c) 2019 John Smith" header in all files when there are hundreds of contributors?

Crop image to path created in TikZ?

Pristine Bit Checking

What do the Banks children have against barley water?



How to Improve Data Scraping Using VBA?


Options for HTML scraping?How to create Excel (.XLS and .XLSX) file in C# without installing Ms Office?Is there a way to crack the password on an Excel VBA Project?How to return a result from a VBA functionHow to avoid using Select in Excel VBAHeadless Browser and scraping - solutionsHow to use Regular Expressions (Regex) in Microsoft Excel both in-cell and loopsData Scraping with td VBAscrape data VBA into excelUsing VBA to scrape data






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;








-2















I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance.
Thanks In Advance



Note-I haven't posted URL as it is clients website. Sorry about that.



Sub FetchData() 
Dim IE As Object
Dim Doc As HTMLDocument
Dim myStr As String
On Error Resume Next

Set IE = CreateObject("InternetExplorer.Application") 'SetBrowser
IE.Visible = False

IE.navigate "URL" 'Open website
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

Set Doc = IE.Document

Doc.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
Doc.getElementById("txtPassword").Value = InputBox("Please Enter Your
Password")
Doc.getElementById("BtnLogin").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

IE.navigate "URL"
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

Dim LastRow As Long

Set wks = ActiveSheet
LastRow = wks.Cells(wks.Rows.Count, "A").End(xlUp).Row
Set rowNo = wks.Range("A1:A" & LastRow)
For rowNo = 2 To LastRow
Doc.getElementById("txtField1").Value =
ThisWorkbook.Sheets("Sheet1").Range("A" & rowNo).Value
Doc.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

strVal1 = Doc.querySelectorAll("span")(33).innerText
ThisWorkbook.Sheets("Sheet1").Range("B" & rowNo).Value = strVal1
strVal2 = Doc.querySelectorAll("span")(35).innerText
ThisWorkbook.Sheets("Sheet1").Range("C" & rowNo).Value = strVal2

Next

End Sub









share|improve this question



















  • 1





    If your code is working, please consider posting to Code Review instead of Stack Overflow

    – Sam
    Mar 8 at 9:00


















-2















I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance.
Thanks In Advance



Note-I haven't posted URL as it is clients website. Sorry about that.



Sub FetchData() 
Dim IE As Object
Dim Doc As HTMLDocument
Dim myStr As String
On Error Resume Next

Set IE = CreateObject("InternetExplorer.Application") 'SetBrowser
IE.Visible = False

IE.navigate "URL" 'Open website
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

Set Doc = IE.Document

Doc.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
Doc.getElementById("txtPassword").Value = InputBox("Please Enter Your
Password")
Doc.getElementById("BtnLogin").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

IE.navigate "URL"
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

Dim LastRow As Long

Set wks = ActiveSheet
LastRow = wks.Cells(wks.Rows.Count, "A").End(xlUp).Row
Set rowNo = wks.Range("A1:A" & LastRow)
For rowNo = 2 To LastRow
Doc.getElementById("txtField1").Value =
ThisWorkbook.Sheets("Sheet1").Range("A" & rowNo).Value
Doc.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

strVal1 = Doc.querySelectorAll("span")(33).innerText
ThisWorkbook.Sheets("Sheet1").Range("B" & rowNo).Value = strVal1
strVal2 = Doc.querySelectorAll("span")(35).innerText
ThisWorkbook.Sheets("Sheet1").Range("C" & rowNo).Value = strVal2

Next

End Sub









share|improve this question



















  • 1





    If your code is working, please consider posting to Code Review instead of Stack Overflow

    – Sam
    Mar 8 at 9:00














-2












-2








-2








I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance.
Thanks In Advance



Note-I haven't posted URL as it is clients website. Sorry about that.



Sub FetchData() 
Dim IE As Object
Dim Doc As HTMLDocument
Dim myStr As String
On Error Resume Next

Set IE = CreateObject("InternetExplorer.Application") 'SetBrowser
IE.Visible = False

IE.navigate "URL" 'Open website
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

Set Doc = IE.Document

Doc.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
Doc.getElementById("txtPassword").Value = InputBox("Please Enter Your
Password")
Doc.getElementById("BtnLogin").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

IE.navigate "URL"
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

Dim LastRow As Long

Set wks = ActiveSheet
LastRow = wks.Cells(wks.Rows.Count, "A").End(xlUp).Row
Set rowNo = wks.Range("A1:A" & LastRow)
For rowNo = 2 To LastRow
Doc.getElementById("txtField1").Value =
ThisWorkbook.Sheets("Sheet1").Range("A" & rowNo).Value
Doc.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

strVal1 = Doc.querySelectorAll("span")(33).innerText
ThisWorkbook.Sheets("Sheet1").Range("B" & rowNo).Value = strVal1
strVal2 = Doc.querySelectorAll("span")(35).innerText
ThisWorkbook.Sheets("Sheet1").Range("C" & rowNo).Value = strVal2

Next

End Sub









share|improve this question
















I have below code, which fetch data from a Intranet. But its taking more time to fetch data.can someone help me to modify the code to increase the performance.
Thanks In Advance



Note-I haven't posted URL as it is clients website. Sorry about that.



Sub FetchData() 
Dim IE As Object
Dim Doc As HTMLDocument
Dim myStr As String
On Error Resume Next

Set IE = CreateObject("InternetExplorer.Application") 'SetBrowser
IE.Visible = False

IE.navigate "URL" 'Open website
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

Set Doc = IE.Document

Doc.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
Doc.getElementById("txtPassword").Value = InputBox("Please Enter Your
Password")
Doc.getElementById("BtnLogin").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

IE.navigate "URL"
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

Dim LastRow As Long

Set wks = ActiveSheet
LastRow = wks.Cells(wks.Rows.Count, "A").End(xlUp).Row
Set rowNo = wks.Range("A1:A" & LastRow)
For rowNo = 2 To LastRow
Doc.getElementById("txtField1").Value =
ThisWorkbook.Sheets("Sheet1").Range("A" & rowNo).Value
Doc.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click
Do While IE.Busy Or IE.ReadyState <> 4
Application.Wait DateAdd("s", 1, Now)
Loop

strVal1 = Doc.querySelectorAll("span")(33).innerText
ThisWorkbook.Sheets("Sheet1").Range("B" & rowNo).Value = strVal1
strVal2 = Doc.querySelectorAll("span")(35).innerText
ThisWorkbook.Sheets("Sheet1").Range("C" & rowNo).Value = strVal2

Next

End Sub






excel vba web-scraping






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Mar 8 at 9:23









Pᴇʜ

25.2k63052




25.2k63052










asked Mar 8 at 7:18









Murali KMurali K

11




11







  • 1





    If your code is working, please consider posting to Code Review instead of Stack Overflow

    – Sam
    Mar 8 at 9:00













  • 1





    If your code is working, please consider posting to Code Review instead of Stack Overflow

    – Sam
    Mar 8 at 9:00








1




1





If your code is working, please consider posting to Code Review instead of Stack Overflow

– Sam
Mar 8 at 9:00






If your code is working, please consider posting to Code Review instead of Stack Overflow

– Sam
Mar 8 at 9:00













1 Answer
1






active

oldest

votes


















2














Can't guarantee this will run. Points to note:



  1. Use of Worksheets collection

  2. Use of Option Explicit - this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range.

  3. Removal of On Error Resume Next

  4. Putting all worksheets into variables

  5. Placement of values into array and looping array to get id values. Looping sheet is expensive

  6. Use of early binding and adding class to InternetExplorer

  7. Assumption that after login a new url present and that you need to navigate back to that before each new loop value

  8. Removal of hungarian notation

  9. Ids are fastest selector method so no improvement there

  10. With your css type selectors, e.g. .document.querySelectorAll("span")(33), you might seek whether there is a single node short selector that can be used, rather than using nodeList

VBA:



Option Explicit 
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet

Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)

With ie

.Visible = False
.Navigate2 "URL" 'Open website

While .Busy Or .readyState < 4: DoEvents: Wend

.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click

While .Busy Or ie.readyState < 4: DoEvents: Wend

Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL

For i = LBound(loopvalues) To UBound(loopvalues)

.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click

While .Busy Or .readyState < 4: DoEvents: Wend

val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2

.Navigate2 newURL

While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub





share|improve this answer

























  • Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?

    – Murali K
    Mar 8 at 9:53












  • You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.

    – QHarr
    Mar 8 at 9:58












  • its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?

    – Murali K
    Mar 8 at 10:28












  • Where i say .navigate2 newUrl

    – QHarr
    Mar 8 at 10:30











  • Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"

    – Murali K
    Mar 8 at 11:41











Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55058469%2fhow-to-improve-data-scraping-using-vba%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









2














Can't guarantee this will run. Points to note:



  1. Use of Worksheets collection

  2. Use of Option Explicit - this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range.

  3. Removal of On Error Resume Next

  4. Putting all worksheets into variables

  5. Placement of values into array and looping array to get id values. Looping sheet is expensive

  6. Use of early binding and adding class to InternetExplorer

  7. Assumption that after login a new url present and that you need to navigate back to that before each new loop value

  8. Removal of hungarian notation

  9. Ids are fastest selector method so no improvement there

  10. With your css type selectors, e.g. .document.querySelectorAll("span")(33), you might seek whether there is a single node short selector that can be used, rather than using nodeList

VBA:



Option Explicit 
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet

Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)

With ie

.Visible = False
.Navigate2 "URL" 'Open website

While .Busy Or .readyState < 4: DoEvents: Wend

.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click

While .Busy Or ie.readyState < 4: DoEvents: Wend

Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL

For i = LBound(loopvalues) To UBound(loopvalues)

.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click

While .Busy Or .readyState < 4: DoEvents: Wend

val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2

.Navigate2 newURL

While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub





share|improve this answer

























  • Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?

    – Murali K
    Mar 8 at 9:53












  • You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.

    – QHarr
    Mar 8 at 9:58












  • its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?

    – Murali K
    Mar 8 at 10:28












  • Where i say .navigate2 newUrl

    – QHarr
    Mar 8 at 10:30











  • Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"

    – Murali K
    Mar 8 at 11:41















2














Can't guarantee this will run. Points to note:



  1. Use of Worksheets collection

  2. Use of Option Explicit - this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range.

  3. Removal of On Error Resume Next

  4. Putting all worksheets into variables

  5. Placement of values into array and looping array to get id values. Looping sheet is expensive

  6. Use of early binding and adding class to InternetExplorer

  7. Assumption that after login a new url present and that you need to navigate back to that before each new loop value

  8. Removal of hungarian notation

  9. Ids are fastest selector method so no improvement there

  10. With your css type selectors, e.g. .document.querySelectorAll("span")(33), you might seek whether there is a single node short selector that can be used, rather than using nodeList

VBA:



Option Explicit 
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet

Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)

With ie

.Visible = False
.Navigate2 "URL" 'Open website

While .Busy Or .readyState < 4: DoEvents: Wend

.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click

While .Busy Or ie.readyState < 4: DoEvents: Wend

Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL

For i = LBound(loopvalues) To UBound(loopvalues)

.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click

While .Busy Or .readyState < 4: DoEvents: Wend

val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2

.Navigate2 newURL

While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub





share|improve this answer

























  • Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?

    – Murali K
    Mar 8 at 9:53












  • You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.

    – QHarr
    Mar 8 at 9:58












  • its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?

    – Murali K
    Mar 8 at 10:28












  • Where i say .navigate2 newUrl

    – QHarr
    Mar 8 at 10:30











  • Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"

    – Murali K
    Mar 8 at 11:41













2












2








2







Can't guarantee this will run. Points to note:



  1. Use of Worksheets collection

  2. Use of Option Explicit - this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range.

  3. Removal of On Error Resume Next

  4. Putting all worksheets into variables

  5. Placement of values into array and looping array to get id values. Looping sheet is expensive

  6. Use of early binding and adding class to InternetExplorer

  7. Assumption that after login a new url present and that you need to navigate back to that before each new loop value

  8. Removal of hungarian notation

  9. Ids are fastest selector method so no improvement there

  10. With your css type selectors, e.g. .document.querySelectorAll("span")(33), you might seek whether there is a single node short selector that can be used, rather than using nodeList

VBA:



Option Explicit 
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet

Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)

With ie

.Visible = False
.Navigate2 "URL" 'Open website

While .Busy Or .readyState < 4: DoEvents: Wend

.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click

While .Busy Or ie.readyState < 4: DoEvents: Wend

Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL

For i = LBound(loopvalues) To UBound(loopvalues)

.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click

While .Busy Or .readyState < 4: DoEvents: Wend

val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2

.Navigate2 newURL

While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub





share|improve this answer















Can't guarantee this will run. Points to note:



  1. Use of Worksheets collection

  2. Use of Option Explicit - this means you then have to use the right datatype throughout. Currently you have undeclared variables and, for example, rowNo is used as a Long and as a range.

  3. Removal of On Error Resume Next

  4. Putting all worksheets into variables

  5. Placement of values into array and looping array to get id values. Looping sheet is expensive

  6. Use of early binding and adding class to InternetExplorer

  7. Assumption that after login a new url present and that you need to navigate back to that before each new loop value

  8. Removal of hungarian notation

  9. Ids are fastest selector method so no improvement there

  10. With your css type selectors, e.g. .document.querySelectorAll("span")(33), you might seek whether there is a single node short selector that can be used, rather than using nodeList

VBA:



Option Explicit 
Public Sub FetchData()
Dim ie As Object, ie As InternetExplorer
Dim lastRow As Long, wks As Worksheet, i As Long, ws As Worksheet

Set ie = New SHDocVw.InternetExplorer 'SetBrowser
Set ws = ThisWorkbook.Worksheets("Sheet1")
Set wks = ActiveSheet '<==use explicit sheet name if possible
lastRow = wks.Cells(wks.rows.Count, "A").End(xlUp).Row
loopvalues = Application.Transpose(wks.Range("A2:A" & lastRow).Value)

With ie

.Visible = False
.Navigate2 "URL" 'Open website

While .Busy Or .readyState < 4: DoEvents: Wend

.document.getElementById("tbxUserID").Value = InputBox("Please Enter Your ID")
.document.getElementById("txtPassword").Value = InputBox("Please Enter Your Password")
.document.getElementById("BtnLogin").Click

While .Busy Or ie.readyState < 4: DoEvents: Wend

Dim newURL As String, val1 As String, val2 As String
newURL = .document.URL

For i = LBound(loopvalues) To UBound(loopvalues)

.document.getElementById("txtField1").Value = loopvalues(i)
.document.getElementById("CtrlQuickSearch1_imgBtnSumbit").Click

While .Busy Or .readyState < 4: DoEvents: Wend

val1 = .document.querySelectorAll("span")(33).innerText
ws.Range("B" & i).Value = val1
val2 = .document.querySelectorAll("span")(35).innerText
ws.Range("C" & i).Value = val2

.Navigate2 newURL

While .Busy Or ie.readyState < 4: DoEvents: Wend
Next
.Quit
End With
End Sub






share|improve this answer














share|improve this answer



share|improve this answer








edited Mar 8 at 15:23

























answered Mar 8 at 8:58









QHarrQHarr

37.4k82244




37.4k82244












  • Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?

    – Murali K
    Mar 8 at 9:53












  • You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.

    – QHarr
    Mar 8 at 9:58












  • its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?

    – Murali K
    Mar 8 at 10:28












  • Where i say .navigate2 newUrl

    – QHarr
    Mar 8 at 10:30











  • Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"

    – Murali K
    Mar 8 at 11:41

















  • Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?

    – Murali K
    Mar 8 at 9:53












  • You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.

    – QHarr
    Mar 8 at 9:58












  • its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?

    – Murali K
    Mar 8 at 10:28












  • Where i say .navigate2 newUrl

    – QHarr
    Mar 8 at 10:30











  • Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"

    – Murali K
    Mar 8 at 11:41
















Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?

– Murali K
Mar 8 at 9:53






Thanks for the replay.. i am getting Run time Error '424' on line ".Document.getElementById("txtField1").Value = loopvalues(i)" and where i should put my 2nd URL in the code? in "newURL = .Document.URL" line?

– Murali K
Mar 8 at 9:53














You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.

– QHarr
Mar 8 at 9:58






You mat need a timed loop (first try adding a wait before that line or stepping through with F8 - does the 424 go away? If not, step through with F8 and check the value of loopValues(i) in locals window.

– QHarr
Mar 8 at 9:58














its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?

– Murali K
Mar 8 at 10:28






its taking A1 value as "loopValues(i)" . actually, after that login page again i have to navigate to the same website with different page which has elements "txtField1" and CtrlQuickSearch1_imgBtnSumbit. in code where i should put the 2nd URL?

– Murali K
Mar 8 at 10:28














Where i say .navigate2 newUrl

– QHarr
Mar 8 at 10:30





Where i say .navigate2 newUrl

– QHarr
Mar 8 at 10:30













Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"

– Murali K
Mar 8 at 11:41





Getting Variable Not Defined on line " ws.Range("B" & rowNo).Value = val1" and Automation Error on line "val1 = .Document.querySelectorAll("span")(33).innerText"

– Murali K
Mar 8 at 11:41



















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f55058469%2fhow-to-improve-data-scraping-using-vba%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Save data to MySQL database using ExtJS and PHP [closed]2019 Community Moderator ElectionHow can I prevent SQL injection in PHP?Which MySQL data type to use for storing boolean valuesPHP: Delete an element from an arrayHow do I connect to a MySQL Database in Python?Should I use the datetime or timestamp data type in MySQL?How to get a list of MySQL user accountsHow Do You Parse and Process HTML/XML in PHP?Reference — What does this symbol mean in PHP?How does PHP 'foreach' actually work?Why shouldn't I use mysql_* functions in PHP?

Compiling GNU Global with universal-ctags support Announcing the arrival of Valued Associate #679: Cesar Manara Planned maintenance scheduled April 23, 2019 at 23:30 UTC (7:30pm US/Eastern) Data science time! April 2019 and salary with experience The Ask Question Wizard is Live!Tags for Emacs: Relationship between etags, ebrowse, cscope, GNU Global and exuberant ctagsVim and Ctags tips and trickscscope or ctags why choose one over the other?scons and ctagsctags cannot open option file “.ctags”Adding tag scopes in universal-ctagsShould I use Universal-ctags?Universal ctags on WindowsHow do I install GNU Global with universal ctags support using Homebrew?Universal ctags with emacsHow to highlight ctags generated by Universal Ctags in Vim?

Add ONERROR event to image from jsp tldHow to add an image to a JPanel?Saving image from PHP URLHTML img scalingCheck if an image is loaded (no errors) with jQueryHow to force an <img> to take up width, even if the image is not loadedHow do I populate hidden form field with a value set in Spring ControllerStyling Raw elements Generated from JSP tagds with Jquery MobileLimit resizing of images with explicitly set width and height attributeserror TLD use in a jsp fileJsp tld files cannot be resolved