-1

I wrote this script for some web scraping, but it isn't working.

I'm a beginner, so I have no idea, also because it isn't showing errors.

I followed this article.

#!/bin/bash

n="1"
rm -f tmp_file extractData.csv
# colunas do arquivo CSV
echo "nome;local;acervo" > livrariasevsp.csv
while [ $n -lt 10 ]
do
    # exec o curl e salva no arquivo temporario
    curl "https://www.estantevirtual.com.br/garimpepor/sebos-e-livreiros/sp?livraria=$n" > tmp_file
    # contador "nome"
    n=$[$n+1]
    # get nome
    nome=$(cat tmp_file |grep "class=\"sebo-nome\"" | cut -d'>' -f4 | cut -d'<' -f1)
    # get local
    local=$(cat tmp_file |grep "class=\"js-sebo-local\"" | grep span | cut -d'>' -f4 | cut -d'<' -f1)
    # get acervo
    acervo=$(cat tmp_file |grep "class=\"js-sebo-acervo\"" | grep span | cut -d'>' -f4 | cut -d'<' -f1)
    # escreve os dados das livrarias no arquivo CSV
    echo "$nome;$local;$acervo" >> livrariasevsp.csv
done
5
  • 3
    What is it supposed to do and what is it doing?
    – jesse_b
    Jun 21, 2019 at 20:33
  • It should take the name of the bookstores on the URL, the city under the name and the number of books. Take a look at the URL and you will see. As a result, I'm just getting the .csv file with the columns, but no data.
    – FabioB
    Jun 21, 2019 at 20:36
  • I corrected the line rm -f tmp_file extractData.csv with rm -f tmp_file livrariasevsp.csv.
    – FabioB
    Jun 21, 2019 at 20:38
  • class="sebo-nome" and class="js-sebo-local" don't exist within that page. class="js-sebo-acervo" exists several times.
    – jesse_b
    Jun 21, 2019 at 20:43
  • On a way I can understand, which classes should I use to grab the information of the first bookstore: "1000 Livros" ; "São Paulo, SP" ; "13.370" ? Because when I inspect these elements, I see the classes.
    – FabioB
    Jun 21, 2019 at 20:46

1 Answer 1

0

I don't usually advocate parsing HTML using regular expressions. However

awk -F'[<>]' '
  BEGIN {print "nome;local;acervo"}
  $2 ~ /^h2 .*-titulo/ {title = $3}
  $2 ~ /^span .*-local/ {city = $3}
  $2 ~ /^span .* nlivros=/ {printf "\"%s\";\"%s\";%d\n", title, city, $3}
' tmp_file > livrariasevsp.csv

Note, the $[...] form of arithmetic expansion is undocumented. Use $((...)) instead. However, you don't actually need to do that. Try this:

{
    echo "nome;local;acervo"
    for n in {1..9}; do
        url="https://www.estantevirtual.com.br/garimpepor/sebos-e-livreiros/sp?livraria=$n"
        awk -F'[<>]' '
            $2 ~ /^h2 .*-titulo/ {title = $3}
            $2 ~ /^span .*-local/ {city = $3}
            $2 ~ /^span .* nlivros=/ {
                printf "\"%s\";\"%s\";%d\n", title, city, $3
            }
        ' <( curl --silent "$url" )
    done 
} > livrariasevsp.csv
8
  • Let the downvoting commence... Jun 21, 2019 at 22:14
  • Although I prefer the answer using curl, so I can understand better, the code worked pretty well. Thanks!
    – FabioB
    Jun 21, 2019 at 23:33
  • This does use curl, see the line above "done" Jun 21, 2019 at 23:44
  • I meant the way I was doing, to understand why my code didn't work.
    – FabioB
    Jun 22, 2019 at 1:01
  • The main fatal flaws: Jun 22, 2019 at 12:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .