README.md 3.73 KB
Newer Older
1
# website\_checks
remy's avatar
remy committed
2

remy's avatar
remy committed
3
4
5
This set of scripts allow some basic tests to check websites :
  - https certificates status,
  - checksums of the index web pages,
6
7
8
9
  - HTTP answers.

`ssl-cert-info.sh` is a modified version from https://gist.github.com/stevenringo/2fe5000d8091f800aee4bb5ed1e800a6 already taken from http://giantdorks.org/alain/shell-script-to-check-ssl-certificate-info-like-expiration-date-and-subject/

10
11
12
13
14
15
16
17
18
19
## How to use

2 main scripts are used here : `check_certs.sh` and `check_urls.sh`.

Each script is looking at a txt file to scan these hosts / urls. Each line is something to check.

First, you will need to enter some informations in `profile.conf` file. This file is sourced by those 2 scripts.

Then, edit `host_https_list.txt` and `url_list.txt` to fit your needs.

remy's avatar
remy committed
20
21
## How to install

22
23
24
25
26
27
28
29
30
31
32
33
### Requirements

Requirements are really basics linux softwares:

  - gnu make,
  - awk & sed,
  - sudo (only needed for a system wide install),
  - curl & wget,
  - sha256sum to perform checksums

___

remy's avatar
remy committed
34
35
36
After editing the 3 configuration files (`profile.conf` and listings `host_https_list.txt`, `url_list.txt`), just run:

```bash
remy's avatar
remy committed
37
sudo make install
remy's avatar
remy committed
38
```
39

remy's avatar
remy committed
40
41
42
43
44
45
Once `websitechecks` is installed, you should find those files here:
  
  - `/etc/default/websitechecks`
  - `/usr/local/websitechecks/etc/host_https_list.txt`
  - `/usr/local/websitechecks/etc/url_list.txt`

remy's avatar
remy committed
46
47
## How to check your HTTPS certificates

remy's avatar
remy committed
48
Edit `host_https_list.txt` file containing the list of hosts (fqdn / fully qualified domain name) to scan.
remy's avatar
remy committed
49
50
51
52

Then, just run:

```bash
remy's avatar
remy committed
53
check_certs.sh
remy's avatar
remy committed
54
55
```

56
This will produce a json output on the standard output and in the output directory (`OUTPUT_DIR` defined in `profile.conf` (default is `/usr/local/websitechecks/results`)).
remy's avatar
remy committed
57

remy's avatar
remy committed
58
59
## How to use checks.sh

remy's avatar
remy committed
60
61
This script checks some URLs (HTTP answers + checksums). Checksums are a good way to check if a website has been defaced. Indeed, any modification on a webpage could e legitiate or not, and this should be monitored.
Note that dynamic webpages are a bit more complicated to monitor by this way. You should remove dynamic element from the webpage before computing the checksum.  
remy's avatar
remy committed
62

remy's avatar
remy committed
63
First, enter the list of URL to scan in `url_list.txt`.
remy's avatar
remy committed
64

remy's avatar
remy committed
65
To initialize the working directory `workdir`, you will need to launch:
remy's avatar
remy committed
66
67

```bash
remy's avatar
remy committed
68
check_urls.sh init
remy's avatar
remy committed
69
70
71
72
73
```

Then, you can do a basic scan:

```bash
remy's avatar
remy committed
74
check_urls.sh check
remy's avatar
remy committed
75
76
```

remy's avatar
remy committed
77
78
You will find your results in `json` format in your `OUTPUT_DIR` defined in `profile.conf`.

remy's avatar
remy committed
79
80
If there is no output on stdout, that is normal. It means that nothing changed since last scan.

remy's avatar
remy committed
81
82
83
To display more informations about differences:

```bash
84
# compare from last run
remy's avatar
remy committed
85
check_urls.sh compare
86
# or check differences since the init step
remy's avatar
remy committed
87
check_urls.sh compare2init
remy's avatar
remy committed
88
89
```

remy's avatar
remy committed
90
Finally, you can add a cron job to produce results every N minutes/days...
remy's avatar
remy committed
91

92
If you detect any changes (see Json files in directory `OUTPUT_DIR`), then, you can also compare it manually:
93
94

```bash
95
cd /usr/local/websitechecks/var
96
97
98
99
100
101
cp -r workdir/ `date +"%Y%m%d"`_workdir
diff -rq <previous_workdir> `date +"%Y%m%d"`_workdir
# then compare specific html pages
diff -Ebw <previous_workdir>/website.index.html `date +"%Y%m%d"`_workdir/website.index.html

# finally, if everything is ok, regenerate a workdir
remy's avatar
remy committed
102
103
check_urls.sh clean
check_urls.sh init
104
105
```

remy's avatar
remy committed
106
After, you will have many directories with all your webpages histories.
107

remy's avatar
remy committed
108
109
110
111
112
113
## Uninstall

```bash
sudo make clean
```

114
## More useful tools
remy's avatar
remy committed
115

116
117
118
Check broken links :
  - http://home.snafu.de/tilman/xenulink.html (need `wine` on Linux)

119
120
121
122
123
124
125
126
127
128
129
Other systems to check HTTPS websites :
  - mixed content : 
    - https://mixed.octopuce.fr

Checking headers :
  - https://securityheaders.com
  - https://observatory.mozilla.org

Tool scan list :
  - https://www.owasp.org/index.php/Category:Vulnerability_Scanning_Tools
  - https://sectools.org/tag/web-scanners/