Ubuntu install Selenium

Ubuntu install Selenium

Most sites can be crawled with simple wget in Bash but some are loaded at runtime using Javascripts. These sites cannot be crawled using simple wget in Bash. You’ll need something like Selenium.

To install Selenium, Firefox and xvfb. Run these commands.

sudo apt update
sudo apt install python-pip firefox xvfb -y
echo export DISPLAY=:99|sudo tee -a /etc/environment
pip install selenium

Download geckodriver. Extract it and move to /usr/bin/

wget https://github.com/mozilla/geckodriver/releases/download/v0.24.0/geckodriver-v0.24.0-linux64.tar.gz
tar -xzvf geckodriver-v0.24.0-linux64.tar.gz
sudo mv geckodriver /usr/bin/

Login as root, and set to run xvfb at reboot.

sudo su
crontab -e
@reboot Xvfb :99 -ac
reboot now

Create a simple python script to test if your installation is successful.

vim test.py
from selenium import webdriver
driver = webdriver.Firefox()
try:
  driver.get('https://www.bing.com')
  print driver.title
finally:
  driver.quit()
python test.py