This is the script :

#!/bin/bash

set -euo pipefail

SOFT="$(basename $0)"
random=$(( ( RANDOM % 100000 ) ))
tempfolder="/tmp/$SOFT/$random"

mkdir -p "$tempfolder"
pushd "$tempfolder" > /dev/null
cleanup ()
 cat wget.txt 

trap cleanup EXIT
site="$1"

wget -nd --spider --recursive --no-verbose --output-file=wget.txt "$site"

the script essentially creates a subfolder in /tmp/crawl and puts into it the result of wget, then feeds it to sed and url_matcher.

Initially I though of putting the cat instruction just after the wget one, but for long wget it didn't work.

The sed instruction removes colons at the end of lines, which is added by wget when said url isn't valid.
(something along the lines of "http://...: Nothing there")

url_matcher is a c++ scanner made with flex, which recognises urls in a text fed with standard input and prints them to standard output, separated with new lines characters.

the scripts allows me to fetch all urls in a website and use that as a stream directly.

Example :

$ crawl www.example.com | sed 's/some_treatments//' > super_file

There's no need to use cat here:

cat wget.txt | sed 's/:$//' | url_matcher

Instead, simply have sed take its input from the file:

sed 's/:$//' wget.txt | url_matcher

Instead of using $RANDOM and mkdir, we can use the mktemp command (GNU coreutils) to create us a new directory (without risk of conflict with an existing one):

tempfolder=$(mktemp -d)

I think that now removes the only Bashism in the script.

Avoid pushd and popd in scripts (having to redirect the unwanted output should be a sign that these are intended for interactive shells). Instead, we can use cd "$tempfolder" and something like cd .. to leave it (it doesn't matter where we end up, as we only use absolute paths after that). Alternatively, don't change directory at all, but specify paths in full (I'll use a shorter name for the temporary directory):

dir=$(mktemp -d)

cleanup() url_matcher
 rm -rf -- "$dir"


trap cleanup EXIT

wget -nd --spider --recursive --no-verbose --output-file="$dir/wget.txt" "$1"

Other alternatives to pushd/popd include saving $PWD into a variable before changing directory, or using cd - (which internally does exactly that, as cd saves $PWD into $OLDPWD for that purpose).

We can avoid the temporary file altogether, to simplify the code and to allow us to start producing output without waiting for the crawl to finish. We can do this in a couple of ways, by being creative with --output-file. If we're on Linux, we can write to /dev/stdout instead of a file:

wget -nd --spider --recursive --no-verbose --output-file=/dev/stdout "$1" 
 | sed -e 's/:$//' | url_matcher

If we're happy to accept a Bash dependency again, we can use process substitution:

wget -nd --spider --recursive --no-verbose 
 --output-file=>(sed -e 's/:$//' | url_matcher) "$1"

However, neither of those dependencies are necessary, as wget follows the common convention of treating the filename - specially, to mean "standard output stream":

wget -nd --spider --recursive --no-verbose --output-file=- "$1" 
 | sed -e 's/:$//' | url_matcher

Modified code

#!/bin/bash

set -uo pipefail

exec wget -nd --spider --recursive --no-verbose --output-file=- "$1" 
 | sed -e 's/:$//' | url_matcher

A few points on top of Toby's excellent review.

This is not correctly protected from word splitting, and will fail when the script's path contains spaces:

SOFT="$(basename $0)"

The only thing you need to double quote there is the $0:

SOFT=$(basename "$0")

Since RANDOM generates a random integer in the range 0 - 32767, the % 100000 is unnecessary.

The popd is unlikely to ever be useful in a cleanup script. Directory changes within a script have no effect outside an executed (as opposed to sourced script). In any case, it's best to not change directories within scripts at all.