Scraping the Web With Python

Good day. So today I was working on a project that I had to do some web scraping. It was, basically, about extracting remote data from web pages, just HTML code, that you have to parse to extract what you need. So having that html code parsed, I had to find and extract some info contained between tables rows, information that was going to be used later in data analisis / data science. And I thought that, maybe, somebody can benefit from the simple and easy to use Python code that I made, well "part of it", to achieve something similar, in the same or different scenarios.

Let's set a simple scenario. Let's say I have a big bunch of images I composed together with Gimp, and used them in my Busy blog. The cat pulled the power cord and y laptop fell, smashed into the ground, and now my mechanical hard drive is dead. Yeah, I know we have SSD now, but anyways, imagine situation :-). Oh no!, I just remembered that I didn't backup these images!, what do I do, what do I do!... Oh, I have these images in my blog. Hmmm, I have two options; either copy the photos from their IPFS links, and expend few minutes saving each photos, one by one, "remember I had and used a big bunch of images", or I can write a super easy Python script that will pull the photos for me. Hmmm, yeah I think it is better to use Python , right? , so let's do it!

So, which python package should I use? Well, I will need to pull remote data from the web, so definitely the best option is to use Requests. Then, I will need to parse that information, in a way that will make it easy to extract what I need. Best option is BeautifulSoup. I will also need to read the image data, which will be in binary, and I will need to work with images, so I will use Pillow to work with the image, and IO to read the data easily into Pillow.

Alright. That's basically it. Now let's code!

Let's Import the Packages first.:

Ok, we have our packages in place. Now let's define a string variable with the URL I want to use to pull information. In this case, I will use my blog post url:

Perfect. Next, a very complicated code!?. Let's pull all html code from that URL into a string variable we will call "html", by using Requests!

Amazing. Now we have our html code! I had to write so much, I need a coffee! But there's still work to do. Now we need to parse our html, so we can then pull all img tags into a python list:

Done! We used the amazing BeautifulSoup package to parse and extract all img tags into a python list, in just 2 lines!

Alright. Now, having all tags in our "images" list variable, we will need to pull the url from the src attribute. So let's declare a new list that will contain all urls, then use a for loop to travel through all items in the list and pull src attribute content using BeautifulSoup's awesome syntax:

Great!, at this point we have all of our ipfs urls as a python list variable. Now, for the final part, we will need to travel the new list, using a for loop in this case, and then make a Requests request to get the content of the url, create a pillow image object by pulling the binary data caught by our request, and because pillow image open method can be a file object in binary mode, we will make use of a binary memory stream, easily, thanks to Python's IO package. Then, we will extract the "hash" from the url, simply using split, and lastly we will save the image as JPEG, keeping the quality at 100%... Let's do it!

And that's it guys! This is the basic code required to scrap my web site, extract the images, and save it locally. If everything goes well (do you have Internet connection? ;-), we will have a list of images named as ".jpg", where is the IPFS hash like this:

Isn't that amazing? How easy it is to do this in Python with the help of all these awesome packages? And you can reuse this over and over... Of course, this is a limited script and you might run into some issues, like when Pillow is not able to recognize or load the image format, in which case you will need to make conditions with conditions using if statements, but nothing crazy. Also there's an IPFS package for python that you can use/implement to figure figure out the real names of the files, and things like that.

If you want to try this code, I wrote and share it publicly on my Repl.it account.

Peace and love to everybody,
And, thanks for reading
Roberto.