Reputation: 21
I need to make a crawler for a tyres website. For that, I need to extract all the dimensions and then pass this dimension in 3 dropdown, click on the search button and extract brand name and price for each product. Then go back to the next set of dimension and..repeat until all the dimensions are finished.
for (int i = 0; i < tyreWidth.size(); i++) {
String option1 = tyreWidth.get(i).getText();
dropdownWidth.selectByIndex(i);
Thread.sleep(3000);
tyreSections = dropdownSection.getOptions();
for (int j = 0; j < tyreSections.size(); j++) {
String option2 = tyreSections.get(j).getText();
dropdownSection.selectByIndex(j);
Thread.sleep(3000);
tyreDiameter = dropdownDiameter.getOptions();
for (int k = 0; k < tyreDiameter.size(); k++) {
dropdownDiameter.selectByIndex(k);
Thread.sleep(3000);
driver.findElement(By.xpath(
".//*[@id='content-groesse']/div[3]/span[2]/button"))
.click();// search button
driver.findElement(By.xpath(
".//*[@id='bottomArticleCount']/ul/li[3]/button"))
.click();// search all oferts
driver.navigate().back();
driver.navigate().back();
Thread.sleep(1000);
listDimensions.add(option1 + " " + option2 + " "
+ tyreDiameter.get(k).getText());
System.out.println(option1 + " " + option2 + " "
+ tyreDiameter.get(k).getText());
}
tyreDiameter.removeAll(tyreDiameter);
}
tyreSections.removeAll(tyreSections);
}
for (int i = 0; i < listDimensions.size(); i++) {
writer.println(listDimensions.get(i));
}
writer.close();
}
Problem: this works only for the first set of dimension. Any sugestions, please? Thanks.
Upvotes: 2
Views: 56
Reputation: 1806
The code for the most part looks fine. I'm going to assume that somewhere higher up is a line similar to:
List<WebElement> tyreWidth = dropdownWidth.getOptions();
Otherwise the initial loop will fail and the whole thing is moot.
The only other suggestion I have is that since the dropdowns are dynamically filling (or at least that's the impression you are giving), the javascript that fills in Selection and Diameter are not being triggered correctly so you always get the defaults. See if this timeline matches up with what you're seeing:
dropdownWidth.selectByIndex(i);
dropdownSection.selectByIndex(j);
dropdownDiameter.selectByIndex(k);
So every time you set the Section, it resets the Width which resets the Section and Diameter is always the default
Try stepping through your code and see if this is the case. If it is, you just need to do something to click out of the dropdown after it is set.
public void clickOut() {
driver.findElement(By.tagName("body")).click();
}
Add clickOut();
after each .selectByIndex()
statement and before the sleep()
. This will cause the dropdown to lose focus and hopefully cause the secondary and tertiary dropdowns to fill properly.
Upvotes: 1