Question Is it possible to regenerate the awstats information based on the https logs in the past?

[danielchau]

Server operating system version

centos 8.4

Plesk version and microupdate number

18.0.30.4

Hello,

I bet your help on this problem,

We are facing a problem that we have a plesk server have been running for years but the awstat page have been only calculated for 4 months statistics, we would like to re-generate the awstats information for 2 years, due to our end user need it.

We are sure that the http logs are still exists in /var/www/vhosts/system/<domain>/logs/access_ssl_log.processed.xxx.gz formats. All the logs are in the log folder COMPRESSED.

Would you please let us know how can we re-generate the awstats information given that all the http logs are inside the server, and peresent them in two years information (which can be selected from the https://domain/plesk-stat/webstat-ssl/ ?

We really need your help on this matter. Thank you.

Daniel

 

[Nikolay]

Hi,
Try to follow steps from this article - Recalculating Statistics for Previous Months

 

[danielchau]

Hello Nikolay,

Thanks for the hint, I am looking at it and I seems able to work with it for webstat-ssl (which the document seems have been just for webstat)

I basically use the following commands.

export vhost_name=mydomain.com

export AWSTATS_BIN_D=`grep ^AWSTATS_BIN_D /etc/psa/psa.conf | awk '{print $2}'`
export HTTPD_VHOSTS_D=`grep ^HTTPD_VHOSTS_D /etc/psa/psa.conf | awk '{print $2}'`
export PRODUCT_ROOT_D=`grep ^PRODUCT_ROOT_D /etc/psa/psa.conf | awk '{print $2}'`
export awstats=${AWSTATS_BIN_D}/awstats.pl
export awstats_gen_opts="-staticlinks -configdir=${PRODUCT_ROOT_D}/etc/awstats -config=${vhost_name}-https"

$awstats $awstats_gen_opts -LogFile=$HTTPD_VHOSTS_D/system/${vhost_name}/statistics/logs/access_ssl_log.processed

for y in 2021 2022 ; do for m in `seq 1 12` ; do mkdir ${HTTPD_VHOSTS_D}/system/${vhost_name}/statistics/webstat-ssl/$y-$(printf "%.2d" $m) ; done ; done


for y in 2021 2022 ; do \
 for m in `seq 1 12` ; do \
     dest_dir=$HTTPD_VHOSTS_D/system/$vhost_name/statistics/webstat-ssl/$y-$(printf "%.2d" $m) ; \
     $awstats $awstats_gen_opts -month=$m -year=$y -output > $dest_dir/awstats.${vhost_name}-https.html ; \
     ln -s $dest_dir/awstats.${vhost_name}-http.html $dest_dir/index.html ; \
     for output in alldomains allhosts lasthosts unknownip allrobots lastrobots session urldetail urlentry urlexit osdetail unknownos refererse refererpages keyphrases keywords errors404 ; do \
         $awstats $awstats_gen_opts -month=$m -year=$y -output=$output > $dest_dir/awstats.${vhost_name}-http.$output.html ; \
     done ; \
 done ; \
done
 $PRODUCT_ROOT_D/admin/sbin/statistics --calculate-one --domain-name=$vhost_name

I am able to create 2 years folders and their txt in /var/www/vhosts/system/mydomain.com/statistics/webstat-ssl .

However I found that after a minute or two, the directory have been reverted back to 4 months folder/txt files situation , and lt looks like the plesk have been regenerated it back.

I can confirm the following PLAN/setting have been in effect:
Service Plans -> Unlimited (Which the domain have been applied to) -> Logs and Statistics -> (tick) Retain web and traffic statistics for 26 months
Tools & settings -> Server Settings -> Retain web and traffic statistics for 26 months

I am not sure what I have missed causing such issue.
Would you please advise?

Thank you
Daniel

 

[danielchau]

Ok...I found out that the problem seems occur just after this command

$PRODUCT_ROOT_D/admin/sbin/statistics --calculate-one --domain-name=$vhost_name

Once I execute it , all things revert back to 4 months folder and txt files in $HTTPD_VHOSTS_D/system/$vhost_name/statistics/webstat-ssl/

..

 

[danielchau]

Hi,
Try to follow steps from this article - Recalculating Statistics for Previous Months

Hello, any idea please?

 

[Peter Debik]

@danielchau You still need to set Tools & Settings > General Settings > Server Settings > "Retain web and traffic statistics for" value to define the number of months that Plesk keeps logs and stats, else Plesk will update its configuration and behavior according to what is set in that field. I assume there is still a "4" in that field? Then change it to 24 to keep in on 2 years.

 

[danielchau]

@danielchau You still need to set Tools & Settings > General Settings > Server Settings > "Retain web and traffic statistics for" value to define the number of months that Plesk keeps logs and stats, else Plesk will update its configuration and behavior according to what is set in that field. I assume there is still a "4" in that field? Then change it to 24 to keep in on 2 years.

Yes, I have also already modified that, but the problem still exist after I run the command
$PRODUCT_ROOT_D/admin/sbin/statistics --calculate-one --domain-name=$vhost_name

...
So I am not sure what have I missed here

 

[Nikolay]

@danielchau Verify that it has been increased in the corresponding service plan, tab Log&Statistics tab

Logs and Statistics

These parameters (located at Service Plans > select a plan > Logs & Statistics tab) define how statistics a...

docs.plesk.com

For existing subscription and custom service plan you can do this using CLI

plesk bin subscription_settings --update example.com -keep_traf_stat  X

X is number of month

https://support.plesk.com/hc/en-us/articles/115001848273-How-to-configure-a-storage-period-for-statistics

 

[danielchau]

Hello @Nikolay @Peter Debik ,

I still have problem regarding your document online:

Recalculating Statistics for Previous Months

By default, Plesk utilities present only the current month’s statistics in HTML format. You can retrieve HTML statist...

docs.plesk.com

In your document, you have this setting:

export awstats_gen_opts="-staticlinks -configdir=${PRODUCT_ROOT_D}/etc/awstats -config=${vhost_name}-http"

So I want https statistics, i changed to

export awstats_gen_opts="-staticlinks -configdir=${PRODUCT_ROOT_D}/etc/awstats -config=${vhost_name}-https"

That's fine

But
In your document

$awstats $awstats_gen_opts -LogFile=$HTTPD_VHOSTS_D/system/${vhost_name}/statistics/logs/access_log.processed

So to change it, it looks like this for SSL logs

$awstats $awstats_gen_opts -LogFile=$HTTPD_VHOSTS_D/system/${vhost_name}/statistics/logs/access_ssl_log.processed

That seems does not work if the domain are already in access_ssl_log.processed.xxx.gz format....

How can we solve this please?

Thanks
Daniel

 

[Peter Debik]

It is correct and expected that rotated logs are no longer available for statistics calculation. Statistics calculation is done before log rotation. In which situation do you need to calculate stats from a rotated log?

 

[danielchau]

Hello @Peter Debik ,

For some reason, we have actually have all the logs on the server, but awstats only show 3 months stats. Now we have to bring back all the stats from the rotated logs.

Thanks
Daniel

 

[Peter Debik]

That case can partially be solved by unzipping the rotated, compressed logs and then applying the solution described in Recalculating Statistics for Previous Months. However, if you have anonymized log files, the last octet of the IP addresses was replaced with a zero on rotation, so statistics may be slightly incorrect regarding returning visitors and similar aspects.